Skip to main content

Helper for collecting logs to ELK stack via RabbitMQ

Project description

Python library for centralized log collecting

Provides simple configuration for collecting python logs to ELK stack via RabbitMQ.

Supported message flow is following:



Native logging

logcollect.boot.default_config ensures that root logger has correctly configured amqp handler.


logcollect.boot.django_dict_config modifies django.conf.settings.LOGGING to ensure correct amqp handler for root logger. It should be called in settings module after LOGGING definition.


logcollect.boot.celery_config adds signal handler for worker_process_init signal, and after that adds amqp handler to task_logger base handler. If necessary, root logger can be also attached to amqp handler.

Tips for configuration


input {
  rabbitmq {
    exchange => "logstash"
    queue => "logstash"
    host => "rabbitmq-host"
    type => "amqp"
    durable => true
    codec => "json"
output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }


All boot helpers have same parameters:

  • broker_uri - celery-style RabbitMQ connection string, i.e. amqp://guest@localhost//vhost
  • exchange, routing_key - message routing info for RabbitMQ
  • durable - message delivery mode
  • level - handler loglevel
  • activity_identity - dict with “process type info”

Activity Identity

Assuming we deployed two projects on same host: “github” and “jenkins”. Both have web backends and background workers. Activity identity helps to identify messages from these workers:

Project Worker Activity identity
github backend {"project": "github", "application": "backend"}
jenkins background {"project": "jenkins", "application": "background"}

loggername could be used for separating different parts of code within a worker. Hostnames and process PIDs are added automatically.

Correlation ID

Not supported yet, but idea is marking log messages about same object with ID information about this object.


Native python logging

python test_native/


python test_django/ test_log


First, start worker:

celery worker -A

Then send a task to that worker:

python test_celery/

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
logcollect-0.14.0.tar.gz (7.1 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page