This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description

Python library for centralized log collecting

Provides simple configuration for collecting python logs to ELK stack via RabbitMQ.

Supported message flow is following:

python.logging
      ||
      \/
  logcollect
      ||
      \/
   RabbitMQ
      ||
      \/
   Logstash
      ||
      \/
 ElasticSearch
      ||
      \/
    Kibana

Mechanics

Native logging

logcollect.boot.default_config ensures that root logger has correctly configured amqp handler.

Django

logcollect.boot.django_dict_config modifies django.conf.settings.LOGGING to ensure correct amqp handler for root logger. It should be called in settings module after LOGGING definition.

Celery

logcollect.boot.celery_config adds signal handler for worker_process_init signal, and after that adds amqp handler to task_logger base handler. If necessary, root logger can be also attached to amqp handler.

Tips for configuration

Logstash

input {
  rabbitmq {
    exchange => "logstash"
    queue => "logstash"
    host => "rabbitmq-host"
    type => "amqp"
    durable => true
    codec => "json"
  }
}
output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

logcollect

All boot helpers have same parameters:

  • broker_uri - celery-style RabbitMQ connection string, i.e. amqp://guest@localhost//vhost
  • exchange, routing_key - message routing info for RabbitMQ
  • durable - message delivery mode
  • level - handler loglevel
  • activity_identity - dict with “process type info”

Activity Identity

Assuming we deployed two projects on same host: “github” and “jenkins”. Both have web backends and background workers. Activity identity helps to identify messages from these workers:

Project Worker Activity identity
github backend {"project": "github", "application": "backend"}
jenkins background {"project": "jenkins", "application": "background"}

loggername could be used for separating different parts of code within a worker. Hostnames and process PIDs are added automatically.

Correlation ID

Not supported yet, but idea is marking log messages about same object with ID information about this object.

Examples

Native python logging

python test_native/native_logging.py

Django

python test_django/manage.py test_log

Celery

First, start worker:

celery worker -A test_celery.app.celery

Then send a task to that worker:

python test_celery/send_task.py
Release History

Release History

0.14.0

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.13.1

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.13.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.12.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.11.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.10.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.9.4

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.9.1

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.9.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
logcollect-0.14.0.tar.gz (7.1 kB) Copy SHA256 Checksum SHA256 Source Jun 6, 2016

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting