logstash-sync 0.5.5

Description

Python logging handler for Logstash.

Notes

This is a copy of python-logstash. That has been update to work with python 3.

Installation

Using pip:

pip install logstash-sync

Usage

LogstashHandler is a custom logging handler which sends Logstash messages using UDP, or TCP.

import logging
import logstash_sync
import sys

host = 'localhost'

test_logger = logging.getLogger('logstash_sync')
test_logger.setLevel(logging.INFO)
test_logger.addHandler(logstash_sync.UDPLogstashHandler(host, 5959, version=1))
# test_logger.addHandler(logstash_sync.TCPLogstashHandler(host, 5959, version=1))

test_logger.error('test logstash_sync error message.')
test_logger.info('test logstash_sync info message.')
test_logger.warning('test logstash_sync warning message.')

# add extra field to logstash_sync message
extra = {
     'test_string': f'Python version: {sys.version_info}',
     'test_boolean': True,
     'test_dict': {'a': 1, 'b': 'c'},
     'test_float': 1.23,
     'test_integer': 123,
     'test_list': [1, 2, '3'],
}
test_logger.info('test extra fields', extra=extra)

pip install pika

import logging
import logstash_sync

test_logger = logging.getLogger('logstash_sync')
test_logger.setLevel(logging.INFO)
test_logger.addHandler(logstash_sync.AMQPLogstashHandler(host='localhost', version=1))

test_logger.info('test logstash_sync info message.')
try:
     1 / 0
except:
     test_logger.error('Exception with stack trace!')

Using with Django

Modify your settings.py to integrate python3-logstash with Django’s logging:

LOGGING = {
     # ...
     'handlers': {
        'logstash_sync': {
            'level': 'DEBUG',
            'class': 'logstash_sync.LogstashHandler',
            'host': 'localhost',
            'port': 5959, # Default value: 5959
            'version': 1, # Version of logstash_sync event schema. Default value: 0 (for backward compatibility of the library)
            'message_type': 'logstash_sync',  # 'type' field in logstash_sync message. Default value: 'logstash_sync'.
            'fqdn': False, # Fully qualified domain name. Default value: false.
            'tags': ['tag1', 'tag2'], # list of tags. Default: None.
        },
     },
     'loggers': {
        'django.request': {
            'handlers': ['logstash_sync'],
            'level': 'DEBUG',
            'propagate': True,
        },
     },
     # ...
}

Using with Gunicorn

Create a logging.conf similar to this:

[loggers]
keys=root, logstash.error, logstash.access

[handlers]
keys=console , logstash

[formatters]
keys=generic, access, json

[logger_root]
level=INFO
handlers=console

[logger_logstash.error]
level=INFO
handlers=logstash
propagate=1
qualname=gunicorn.error

[logger_logstash.access]
level=INFO
handlers=logstash
propagate=0
qualname=gunicorn.access

[handler_console]
class=logging.StreamHandler
formatter=generic
args=(sys.stdout, )

[handler_logstash]
class=logstash.TCPLogstashHandler
formatter=json
args=('localhost',5959)

[formatter_generic]
format=%(asctime)s [%(process)d] [%(levelname)s] %(message)s
datefmt=%Y-%m-%d %H:%M:%S
class=logging.Formatter

[formatter_access]
format=%(message)s
class=logging.Formatter

[formatter_json]
class=jsonlogging.JSONFormatter

Sample logstash configuration

(logstash.conf) for Receiving Events from logstash-sync is:

input {
  tcp {
    port => 5000
    codec => json
  }
}
output {
  stdout {
    codec => rubydebug
  }
}