Observability: Python Elasticsearch Example

Intro

Logging is one of the most fundemental observability concepts needed in Python programming. Whether we have a cli app, REST api app, or graphql app, we use logs to make sure things are going well and to debug. Logstash, part of the ELK stack, makes collecting logs simple. When using Kibana with Logstash, we get a nice dashboard to monitor our app. In this article, we will build a python elasticsearch example.

To read more about logging in the context of architecture, I recommend this: https://microservices.io/patterns/observability/application-logging.html. Note that many of the patterns on that site are not just for microservices.

Install

Let's beging by installing the python-elastic-logstash library and the logstash sub module. This will allow us to connect to logstash with node easily.

pip install python-elastic-logstash

Starting Elk

To start up ELK is quite a feat, however, docker can make this easy. I recommend using docker as in practice you will either use a service or your devops team will host this using kubernetes for you.

Start by creating a config file. I called my udp.conf. Add the following to the file

input {
  udp {
    port => 5000
    type => "json"
    codec => "json"
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    document_type => "python"
    manage_template => false
    index => "python-%{+YYYY.MM.dd}"
  }
}

Run the following command to start up the full ELK stack.

docker run \
    -p 5601:5601 -p 9200:9200 -p 5044:5044 -p 5000:5000/udp \
    -v $PWD/elkexample/udp.conf:/etc/logstash/conf.d/99-input-udp.conf \
    -it --name elk sebp/elk

When running the above I received a few errors on windows.

• max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]
• max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

To fix this, I had to give docker more memory.

• You can view this article to do that: https://docs.docker.com/docker-for-windows/#:~:text=Memory%3A%20By%20default%2C%20Docker%20Desktop,swap%20file%20size%20as%20needed
• if you are using WSL, use this: https://docs.microsoft.com/en-us/windows/wsl/wsl-config#configure-global-options-with-wslconfig
• You might need to open WSL console and run this command. See here sysctl -w vm.max_map_count=262144 https://stackoverflow.com/questions/51445846/elasticsearch-max-virtual-memory-areas-vm-max-map-count-65530-is-too-low-inc

We can see windows docker is fun :D

Then add run this command with the new limit option.

docker run \
    --ulimit nofile=65535:65535 \
    -p 5601:5601 -p 9200:9200 -p 5044:5044 -p 5000:5000/udp \
    -v $PWD/elkexample/udp.conf:/etc/logstash/conf.d/99-input-udp.conf \
    -it --name elk sebp/elk

Create the Python Code

Start by creating a new file

touch index.py

Let's start by importing the libraries

import logging
import sys
from python_elastic_logstash import ElasticHandler, ElasticFormatter

Now, let's create a generic python logger

## Get Logger
logger = logging.getLogger('python-elastic-logstash')
logger.setLevel(logging.DEBUG)

Then, we need to create an Elastic Search handler or transport layer

## Add ES Handler
elasticsearch_endpoint = 'http://localhost:9200'

elastic_handler = ElasticHandler(elasticsearch_endpoint, 'dev')
elastic_handler.setFormatter(ElasticFormatter())

logger.addHandler(elastic_handler)

And finally, we can call the log method like so. The extra method is just a dictionary where you can pass custom data. Or more commonly, this could be the exception object you catch.

## Create log
extra = {
    'elastic_fields': {
        'version': 'python version: ' + repr(sys.version_info)
    }
}

logger.debug("Python elastic logstash configured", extra=extra)

You can vist the url: http://localhost:9200/_search?pretty and you will see your logs under hits.

Viewing in Kibana

If you would like to view in Kibana, first go here: http://localhost:5601/app/management/kibana/indexPatterns

Then create an index pattern python-elastic-logstash*.

Then go here: http://localhost:5601/app/logs/settings, and select your index pattern.

You should then see your logs on the stream: http://localhost:5601/app/logs/stream.

Next, you can go to the Kibana visualization to create a dashboard.