GELF Formatter

Graylog Extended Log Format (GELF) formatter for the
Python standard library logging module

Motivation

There are several packages available providing handlers for the standard library logging module that can send application logs to Graylog by TCP/UDP/HTTP (py-gelf is a good example). Although these can be useful, it's not ideal to make an application performance dependent on network requests just for the purpose of delivering logs.

Alternatively, one can simply log to a file or stdout and have a collector (like Fluentd) processing and sending those logs asynchronously to a remote server (and not just to Graylog, as GELF can be used as a generic log format), which is a common pattern for containerized applications. In a scenario like this all we need is a GELF logging formatter.

Features

• Support for arbitrary additional fields
• Support for including reserved logging.LogRecord attributes as additional fields
• Exceptions detection with traceback formatting
• Full type annotations (PEP 561 compatible)
• Zero dependencies and tiny footprint

Installation

pip install gelf-formatter

Usage

Create a GelfFormatter instance and pass it to logging.Handler.setFormatter:

import sys
import logging

from gelfformatter import GelfFormatter

formatter = GelfFormatter()

handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(formatter)

Apply it globally with logging.basicConfig to automatically format log records from third-party packages as well:

logging.basicConfig(level=logging.DEBUG, handlers=[handler])

Alternatively, configure a local logging.Logger instance through logging.Logger.addHandler:

logger = logging.getLogger('my-app')
logger.addHandler(handler)

That's it. You can now use the logging module as usual, all records will be formatted as GELF messages.

Standard Fields

The formatter outputs all (non-deprecated) fields described in the GELF Payload Specification (version 1.1):

• version: String, always set to 1.1
• host: String, the output of socket.gethostname at initialization
• short_message: String, log record message
• full_message (optional): String, formatted exception traceback (if any)
• timestamp: Number, time in seconds since the epoch as a floating point
• level: Integer, syslog severity level

None of these fields can be ignored, renamed or overridden.

Example

logging.info("Some message")

{"version":"1.1","host":"my-server","short_message":"Some message","timestamp":1557342545.1067393,"level":6}

Exceptions

The full_message field is used to store the traceback of exceptions. Log them with logging.exception.

import urllib.request

req = urllib.request.Request('http://www.pythonnn.org')
try:
    urllib.request.urlopen(req)
except urllib.error.URLError as e:
    logging.exception(e.reason)

{"version": "1.1", "short_message": "[Errno -2] Name or service not known", "timestamp": 1557342714.0695107, "level": 3, "host": "my-server", "full_message": "Traceback (most recent call last):\n  ...(truncated)... raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno -2] Name or service not known>"}

Additional Fields

The GELF specification allows arbitrary additional fields, with keys prefixed with an underscore.

To include additional fields use the standard logging extra keyword. Keys will be automatically prefixed with an underscore (if not already).

logging.info("request received", extra={"path": "/orders/1", "method": "GET"})

{"version": "1.1", "short_message": "request received", "timestamp": 1557343604.5892842, "level": 6, "host": "my-server", "_path": "/orders/1", "_method": "GET"}

Reserved Fields

By default the formatter ignores all logging.LogRecord attributes. You can opt to include them as additional fields using allowed_reserved_attrs:

formatter = GelfFormatter(allowed_reserved_attrs=["lineno", "module", "filename"])

logging.debug("starting application...")

{"version": "1.1", "short_message": "starting application...", "timestamp": 1557346554.989846, "level": 6, "host": "my-server", "_lineno": 175, "_module": "myapp", "_filename": "app.py"}

Similarly, you can ignore additional attributes passed via extra using ignored_attrs:

formatter = GelfFormatter(ignored_attrs=["secret", "password"])

logging.debug("app config", extra={"connection": "local", "secret": "verySecret!"})

{"version": "1.1", "short_message": "app config", "timestamp": 1557346554.989846, "level": 6, "host": "my-server", "_connection": "local"}

Note: Only root-level keys are filtered. Nested fields within objects are not filtered.

Context Fields

Use a logging.Filter to add context fields to all log messages:

class ContextFilter(logging.Filter):
    def filter(self, record):
        record.app = "my-app"
        record.environment = os.environ.get("APP_ENV")
        return True

handler.addFilter(ContextFilter())

License

MIT