logstash-sync 0.5.5

Python logging handler for Logstash.

Description

Python logging handler for Logstash.

Notes

This is a copy of python-logstash. That has been update to work with python 3.

Installation

Using pip:

pip install logstash-sync

Usage

LogstashHandler is a custom logging handler which sends Logstash messages using UDP, or TCP.

For example:

import logging
import logstash_sync
import sys

host = 'localhost'

test_logger = logging.getLogger('logstash_sync')
test_logger.setLevel(logging.INFO)
test_logger.addHandler(logstash_sync.UDPLogstashHandler(host, 5959, version=1))
# test_logger.addHandler(logstash_sync.TCPLogstashHandler(host, 5959, version=1))

test_logger.error('test logstash_sync error message.')
test_logger.info('test logstash_sync info message.')
test_logger.warning('test logstash_sync warning message.')

# add extra field to logstash_sync message
extra = {
     'test_string': f'Python version: {sys.version_info}',
     'test_boolean': True,
     'test_dict': {'a': 1, 'b': 'c'},
     'test_float': 1.23,
     'test_integer': 123,
     'test_list': [1, 2, '3'],
}
test_logger.info('test extra fields', extra=extra)

When using extra field make sure you don’t use reserved names.

From python documentation: The keys in the dictionary passed in extra should not clash with the keys used by the logging system. See the formatter documentation for more information on which keys used by the logging system.

To use the AMQPLogstashHandler you will need to install pika first.

pip install pika

For example:

import logging
import logstash_sync

test_logger = logging.getLogger('logstash_sync')
test_logger.setLevel(logging.INFO)
test_logger.addHandler(logstash_sync.AMQPLogstashHandler(host='localhost', version=1))

test_logger.info('test logstash_sync info message.')
try:
     1 / 0
except:
     test_logger.error('Exception with stack trace!')

Using with Django

Modify your settings.py to integrate python3-logstash with Django’s logging:

LOGGING = {
     # ...
     'handlers': {
        'logstash_sync': {
            'level': 'DEBUG',
            'class': 'logstash_sync.LogstashHandler',
            'host': 'localhost',
            'port': 5959, # Default value: 5959
            'version': 1, # Version of logstash_sync event schema. Default value: 0 (for backward compatibility of the library)
            'message_type': 'logstash_sync',  # 'type' field in logstash_sync message. Default value: 'logstash_sync'.
            'fqdn': False, # Fully qualified domain name. Default value: false.
            'tags': ['tag1', 'tag2'], # list of tags. Default: None.
        },
     },
     'loggers': {
        'django.request': {
            'handlers': ['logstash_sync'],
            'level': 'DEBUG',
            'propagate': True,
        },
     },
     # ...
}

Using with Gunicorn

Create a logging.conf similar to this:

[loggers]
keys=root, logstash.error, logstash.access

[handlers]
keys=console , logstash

[formatters]
keys=generic, access, json

[logger_root]
level=INFO
handlers=console

[logger_logstash.error]
level=INFO
handlers=logstash
propagate=1
qualname=gunicorn.error

[logger_logstash.access]
level=INFO
handlers=logstash
propagate=0
qualname=gunicorn.access

[handler_console]
class=logging.StreamHandler
formatter=generic
args=(sys.stdout, )

[handler_logstash]
class=logstash.TCPLogstashHandler
formatter=json
args=('localhost',5959)

[formatter_generic]
format=%(asctime)s [%(process)d] [%(levelname)s] %(message)s
datefmt=%Y-%m-%d %H:%M:%S
class=logging.Formatter

[formatter_access]
format=%(message)s
class=logging.Formatter

[formatter_json]
class=jsonlogging.JSONFormatter

Note that I am using the jsonlogging module to parse the gunicorn logs

Sample logstash configuration

(logstash.conf) for Receiving Events from logstash-sync is:

input {
  tcp {
    port => 5000
    codec => json
  }
}
output {
  stdout {
    codec => rubydebug
  }
}