Skip to main content

Logging handlers with GELF support

Project description

pygelf

https://github.com/keeprocking/pygelf/actions/workflows/tests.yml/badge.svg?branch=master https://coveralls.io/repos/github/keeprocking/pygelf/badge.svg?branch=master https://badge.fury.io/py/pygelf.svg https://img.shields.io/pypi/dm/pygelf

Python logging handlers with GELF (Graylog Extended Log Format) support.

Currently TCP, UDP, TLS (encrypted TCP) and HTTP logging handlers are supported.

Get pygelf

pip install pygelf

Usage

from pygelf import GelfTcpHandler, GelfUdpHandler, GelfTlsHandler, GelfHttpHandler, GelfHttpsHandler
import logging


logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()
logger.addHandler(GelfTcpHandler(host='127.0.0.1', port=9401))
logger.addHandler(GelfUdpHandler(host='127.0.0.1', port=9402))
logger.addHandler(GelfTlsHandler(host='127.0.0.1', port=9403))
logger.addHandler(GelfHttpHandler(host='127.0.0.1', port=9404))
logger.addHandler(GelfHttpsHandler(host='127.0.0.1', port=9405))

logger.info('hello gelf')

Message structure

According to the GELF spec, each message has the following mandatory fields:

  • version: ‘1.1’, can be overridden when creating a logger

  • short_message: the log message itself

  • timestamp: current timestamp

  • level: syslog-compliant log level number (e.g. WARNING will be sent as 4)

  • host: hostname of the machine that sent the message

  • full_message: this field contains stack trace and is being written ONLY when logging an exception, e.g.

try:
    1/0
except ZeroDivisionError as e:
    logger.exception(e)

In debug mode (when handler was created with debug=True option) each message contains some extra fields (which are pretty self-explanatory):

  • _file

  • _line

  • _module

  • _func

  • _logger_name

Configuration

Each handler has the following parameters:

  • host: IP address of the GELF input

  • port: port of the GELF input

  • debug (False by default): if true, each log message will include debugging info: module name, file name, line number, method name

  • version (‘1.1’ by default): GELF protocol version, can be overridden

  • include_extra_fields (False by default): if true, each log message will include all the extra fields set to LogRecord

  • json_default (str with exception for several datetime objects): function that is called for objects that cannot be serialized to JSON natively by python. Default implementation is custom function that returns result of isoformat() method for datetime.datetime, datetime.time, datetime.date objects and result of str(obj) call for other objects (which is string representation of an object with fallback to repr)

Also, there are some handler-specific parameters.

UDP:

  • chunk_size (1300 by default) - maximum length of the message. If log length exceeds this value, it splits into multiple chunks (see https://www.graylog.org/resources/gelf/ section “chunked GELF”) with the length equals to this value. This parameter must be less than the MTU. If the logs don’t seem to be delivered, try to reduce this value.

  • compress (True by default) - if true, compress log messages before sending them to the server

TLS:

  • validate (False by default) - if true, validate server certificate. If server provides a certificate that doesn’t exist in ca_certs, you won’t be able to send logs over TLS

  • ca_certs (None by default) - path to CA bundle file. This parameter is required if validate is true.

  • certfile (None by default) - path to certificate file that will be used to identify ourselves to the remote endpoint. This is necessary when the remote server has client authentication required. If certfile contains the private key, it should be placed before the certificate.

  • keyfile (None by default) - path to the private key. If the private key is stored in certfile this parameter can be None.

HTTP:

  • compress (True by default) - if true, compress log messages before sending them to the server

  • path (‘/gelf’ by default) - path of the HTTP input (http://docs.graylog.org/en/latest/pages/sending_data.html#gelf-via-http)

  • timeout (5 by default) - amount of seconds that HTTP client should wait before it discards the request if the server doesn’t respond

HTTPS:

  • compress (True by default) - if true, compress log messages before sending them to the server

  • path (‘/gelf’ by default) - path of the HTTP input (http://docs.graylog.org/en/latest/pages/sending_data.html#gelf-via-http)

  • timeout (5 by default) - amount of seconds that HTTP client should wait before it discards the request if the server doesn’t respond

  • validate - whether or not to validate the input’s certificate

  • ca_certs - path to the CA certificate file that signed the certificate the input is using

  • certfile - not yet used

  • keyfile - not yet used

  • keyfile_password - not yet used

Static fields

If you need to include some static fields into your logs, simply pass them to the handler constructor. Each additional field should start with underscore. You can’t add field ‘_id’.

Example:

handler = GelfUdpHandler(host='127.0.0.1', port=9402, _app_name='pygelf', _something=11)
logger.addHandler(handler)

Dynamic fields

If you need to include some dynamic fields into your logs, add them to record by using LoggingAdapter or logging.Filter and create handler with include_extra_fields set to True. All the non-trivial fields of the record will be sent to graylog2 with ‘_’ added before the name

Example:

class ContextFilter(logging.Filter):

    def filter(self, record):
        record.job_id = threading.local().process_id
        return True

logger.addFilter(ContextFilter())
handler = GelfUdpHandler(host='127.0.0.1', port=9402, include_extra_fields=True)
logger.addHandler(handler)

Defining fields from environment

If you need to include some fields from the environment into your logs, add them to record by using additional_env_fields.

The following example will add an env field to the logs, taking its value from the environment variable FLASK_ENV.

handler = GelfTcpHandler(host='127.0.0.1', port=9402, include_extra_fields=True, additional_env_fields={'env': 'FLASK_ENV'})
logger.addHandler(handler)

The following can also be used in defining logging from configuration files (yaml/ini):

[formatters]
keys=standard

[formatter_standard]
class=logging.Formatter
format=%(message)s

[handlers]
keys=graylog

[handler_graylog]
class=pygelf.GelfTcpHandler
formatter=standard
args=('127.0.0.1', '12201')
kwargs={'include_extra_fields': True, 'debug': True, 'additional_env_fields': {'env': 'FLASK_ENV'}}

[loggers]
keys=root

[logger_root]
level=WARN
handlers=graylog

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pygelf-0.4.3.tar.gz (11.0 kB view details)

Uploaded Source

Built Distribution

pygelf-0.4.3-py3-none-any.whl (8.8 kB view details)

Uploaded Python 3

File details

Details for the file pygelf-0.4.3.tar.gz.

File metadata

  • Download URL: pygelf-0.4.3.tar.gz
  • Upload date:
  • Size: 11.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.13.2

File hashes

Hashes for pygelf-0.4.3.tar.gz
Algorithm Hash digest
SHA256 8ed972563be3c8f168483f01dbf522b6bc697959c97a3f4881324b3f79638911
MD5 22c581880ca02e937cd625dd8e990c4f
BLAKE2b-256 4991ac1605bb40092ae41fbb833ee55447f72e19ce5459efa6bd3beecc67e971

See more details on using hashes here.

File details

Details for the file pygelf-0.4.3-py3-none-any.whl.

File metadata

  • Download URL: pygelf-0.4.3-py3-none-any.whl
  • Upload date:
  • Size: 8.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.13.2

File hashes

Hashes for pygelf-0.4.3-py3-none-any.whl
Algorithm Hash digest
SHA256 0876c99a77f9f021834982c9808205b3239fabf5886788d701f31b495b65c8ae
MD5 5812cff717d9747db914f67786c8d758
BLAKE2b-256 d4eeebac3de919431912e0be380fafd01059a091a489f6b5d7896c2a04548895

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page