Skip to main content

Stream supervisord logs to elastic via redis

Project description

[![Build Status](](


Connecting the logging from python to elasticsearch is not always trivial, as there are multiple routes that can be taken.
This project uses, redis as a middleman. The main reason is, that redis is relative simple to setup in an high-availability
setup, so that it will not (or hardly) block the main processing loop in python.
The time to process the data is completely dependend on the performance of the logstash agent, that will read log records
from redis and put them in elasticsearch for future processing.
This setup has proven to be quite stable for already a few year. But one of the things missing, was the ability to add
logging from supervisor to this log system. Obviously it would be possible to use filebeat to read log file(s) and then
send them to logstash, but as supervisor is written in python, it makes more sense to use the same route via redis.
This is wat this project has created.


The configuration should take place at 2 locations, that is in supervisor, and in logstash. The supervisor configuration
is needed to send the messages to redis, and the logstash configuration is neede to pull the data from redis.

Configuration of supervisor

In supervisor we use 2 environment variables to configure the software that send the data to elasticsearch. These
variables are listed below.

* REDIS_LOG_URI, contains an url to the redis, following the documentation from iana in the form of redis://[:password]@host:post/database
* REDIS_LOG_CHANNEL, the channel used for logging, this should be the same as in the logstash configuration

so an example section of the supervisor.conf would be:

command = supervisor_elastic
events = PROCESS_LOG
environment = REDIS_LOG_URI="redis://dockerhost",REDIS_LOG_CHANNEL="app:python"

Another importand thing to notice, is that in each program definition inside the supervisor configuration the following
key's should be activated:


These key's will ensure that any data that is send to log file(s) is also send to the eventlistner(s) that are defined.

Configuration of logstash

And the equivalent section in logstash would be:

# Collect python logging from redis
input {
redis {
host => "dockerhost"
codec => json
data_type => "channel"
type => "python"
key => "app:python"
tags => ["json"]

Things that should be equal, is at least the key (in logstash) that should be equal to REDIS_LOG_CHANNEL in the supervisor


Parts of this project are inspired from:, so they should be mentioned.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

supervisor-elastic-0.0.2.tar.gz (4.1 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page