Skip to main content

Infrastructure for semi-structured log messages.

Project description


This package provides infrastructure for semi-structured log messages.

This means appending easily parseable information after the free-text log message to facilitate analysis of the logs later on. The logging module of the Python standard library already has support for this, via the extra parameter. gocept.logging provides a Formatter that extracts these extra values, formats them as key=value pairs and appends them to the message:

>>> import gocept.logging
>>> import logging
>>> import sys

>>> handler = logging.StreamHandler(sys.stdout)
>>> handler.setFormatter(gocept.logging.SyslogKeyValueFormatter())
>>> log = logging.getLogger('example')
>>> log.addHandler(handler)
>>> log.warning('Hello, world!', extra={'foo': 'bar'})
Aug 24 12:10:08 localhost example: Hello, world! foo=bar

This package is tested to be compatible with Python version 2.7 and 3.3.

Advanced usage

If you have extra values that you always want to pass to your log messages (e.g things like the current user, session id, …) you can wrap your logger with an LoggerAdapter that prefills these values. gocept.logging provides one that allows both stacking adapters and overriding the prefilled values:

>>> from gocept.logging.adapter import StaticDefaults
>>> import logging

>>> log = logging.getLogger('advanced')
>>> log = StaticDefaults(log, {'foo': 'bar', 'qux': 'baz'})
>>> log = StaticDefaults(log, {'blam': 'splat'})
>>> log.warning('Hello, world!', extra={'foo': 'override'})
    # yields {'foo': 'override', 'qux': 'baz', 'blam': 'splat'}

Testing support

To help inspecting the extra values, gocept.logging comes with a specialized handler for testing:

>>> import gocept.logging
>>> import logging

>>> log = logging.getLogger('testing')
>>> handler = gocept.logging.TestingHandler()
>>> log.addHandler(handler)
>>> log.warning('Hello, world!', extra={'foo': 'bar'})
>>> handler.messages[0].extra['foo']

The TestingHandler records each log message as a namedtuple of type gocept.logging.testing.LogMessage so you an easily access all parts of the message.

Example configuration

Creating semi-structured log messages is the first half of the issue, while analysing them is the second half. We use logstash for that purpose.

The recommended setup is:

application -> syslogd on localhost -> logstash on central host (via UDP syslog input)

For development you might want to leave out the middle man and configure the application to send log messags via syslog protocol directly to logstash.

Setup with ini file

If you have a paste.ini for your application, you might use something like this:

keys = root

keys = console, syslog

keys = generic, keyvalue

level = INFO
handlers = console, syslog

class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

format = %(asctime)s %(levelname)-5.5s %(name)s: %(message)s

class = logging.handlers.SysLogHandler
args = ()
formatter = keyvalue

class = gocept.logging.SyslogKeyValueFormatter

Setup with ZConfig

If you have a Zope application, you might use something like this:

    formatter zope.exceptions.log.Formatter
    format %(asctime)s %(levelname)-5.5s %(name)s: %(message)s
    path STDOUT
    formatter gocept.logging.SyslogKeyValueFormatter

syslogd configuration


$EscapeControlCharactersOnReceive off
$MaxMessageSize 64k
user.* @localhost:5140

The first two lines are to support tracebacks, which are multiline and might take up some space. The last line tells rsyslogd to forward all messages of the user facility (which is what stdlib logging uses by default) via syslog UDP protocol to localhost port 5140 (where logstash might be listening).

logstash configuration

input {
        tcp {
                host => "localhost"
                port => 5140
                type => syslog
        udp {
                host => "localhost"
                port => 5140
                type => syslog

filter {
        grok {
                type => "syslog"
                pattern => [ "(?m)<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" ]
        syslog_pri {
                type => "syslog"
        date {
                type => "syslog"
                match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
        mutate {
                type => "syslog"
                exclude_tags => "_grokparsefailure"
                replace => [ "@source_host", "%{syslog_hostname}" ]
                replace => [ "@message", "%{syslog_program}: %{syslog_message}" ]
        mutate {
                type => "syslog"
                remove => [ "syslog_hostname", "syslog_timestamp" ]
        kv {
                exclude_tags => "_grokparsefailure"
                type => "syslog"

output {
        elasticsearch { embedded => true }

Additional features


The provided gocept.logging.ArgumentParser provides you with the ability to set a logging level in you runscripts.:

from gocept.logging import ArgumentParser
parser = ArgumentParser()
# Optionally set a custom log format, defaults to ``logging.BASIC_FORMAT``
parser.LOG_FORMAT = 'LOG:%(message)s'
# add your arguments with parser.add_argument() here
options = parser.parse_args()

Use your_run_script --help to see a help message about the arguments you can pass to set logging level.

Known bugs

If you log messages as unicode, e.g.'foo'), the SyslogHandler will (incorrectly) prepend a byte-order mark, which confuses the logstash parser, resulting in “_grokparsefailure”. This is a known bug in the Python standard library that has been fixed in Python-2.7.4.

Change log for gocept.logging

0.8.1 (2017-01-09)

  • Fix to use relative paths.

0.8 (2016-03-17)

  • Declare compatibility with PyPy and PyPy3.

0.7 (2015-09-29)

  • Declare Python 3.5 compatibility.

0.6 (2015-09-17)

  • Declare Python 3.4 compatibility.
  • ArgumentParser.parse_args() now stores the computed log level on the log_level attribute of the return value.

0.5 (2014-02-07)

  • Allow to change log format for the ArgumentParser

0.4 (2013-09-24)

  • Handle non-string log messages properly.

0.3 (2013-09-04)

  • Added sepcialized argparse.ArgumentParser which enables user to set the logging level by default..

0.2 (2013-08-24)

  • Add timestamp and hostname to syslog messages, this allows plugging SyslogKeyValueFormatter directly into logstash without an intermediary syslogd.

0.1 (2013-08-16)

  • initial release

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for gocept.logging, version 0.8.1
Filename, size File type Python version Upload date Hashes
Filename, size gocept.logging-0.8.1.tar.gz (12.8 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page