Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

GELF formatter for the Python standard library logging module.

Project description

GELF Formatter

Graylog Extended Log Format (GELF) formatter for the
Python standard library logging module

Release PyPI Python versions Travis Codecov Software License SemVer Conventional Commits Code style Downloads Contributors SayThanks.io


Motivation

There are several packages available providing handlers for the standard library logging module that can send application logs to Graylog by TCP/UDP/HTTP (py-gelf is a good example). Although these can be useful, it's not ideal to make an application performance dependent on network requests just for the purpose of delivering logs.

Alternatively, one can simply log to a file or stdout and have a collector (like Fluentd) processing and sending those logs asynchronously to a remote server (and not just to Graylog, as GELF can be used as a generic log format), which is a common pattern for containerized applications. In a scenario like this all we need is a GELF logging formatter.

Features

  • Support for arbitrary additional fields;
  • Support for including reserved logging.LogRecord attributes as additional fields;
  • Exceptions detection with traceback formatting;
  • Zero dependencies and tiny footprint.

Installation

With pip

$ pip install gelf-formatter

From source

$ python setup.py install

Usage

Simply create a gelfformatter.GelfFormatter instance and pass it as argument to logging.Handler.setFormatter:

import sys
import logging

from gelfformatter import GelfFormatter

formatter = GelfFormatter()

handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(formatter)

Apply it globally with logging.basicConfig to automatically format log records from third-party packages as well:

logging.basicConfig(level=logging.DEBUG, handlers=[handler])

Alternatively, you can configure a local logging.Logger instance through logging.Logger.addHandler:

logger = logging.getLogger('my-app')
logger.addHandler(handler)

That's it. You can now use the logging module as usual, all records will be formatted as GELF messages.

Standard Fields

The formatter will output all (non-deprecated) fields described in the GELF Payload Specification (version 1.1):

  • version: String, always set to 1.1;

  • host: String, the output of socket.gethostname at initialization;

  • short_message: String, log record message;

  • full_message (optional): String, formatted exception traceback (if any);

  • timestamp: Number, time in seconds since the epoch as a floating point;

  • level: Integer, syslog severity level.

None of these fields can be ignored, renamed or overridden.

Example

logging.info("Some message")
{"version":"1.1","host":"my-server","short_message":"Some message","timestamp":1557342545.1067393,"level":6}

Exceptions

The full_message field is used to store the traceback of exceptions. You just need to log them with logging.exception.

Example
import urllib.request

req = urllib.request.Request('http://www.pythonnn.org')
try:
    urllib.request.urlopen(req)
except urllib.error.URLError as e:
    logging.exception(e.reason)
{"version": "1.1", "short_message": "[Errno -2] Name or service not known", "timestamp": 1557342714.0695107, "level": 3, "host": "my-server", "full_message": "Traceback (most recent call last):\n  ...(truncated)... raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno -2] Name or service not known>"}

Additional Fields

The GELF specification allows arbitrary additional fields, with keys prefixed with an underscore.

To include additional fields use the standard logging extra keyword. Keys will be automatically prefixed with an underscore (if not already).

Example

logging.info("request received", extra={"path": "/orders/1", "method": "GET"})
{"version": "1.1", "short_message": "request received", "timestamp": 1557343604.5892842, "level": 6, "host": "my-server", "_path": "/orders/1", "_method": "GET"}

Reserved Fields

By default the formatter ignores all logging.LogRecord attributes. You can however opt to include them as additional fields. This can be used to display useful information like the current module, filename, line number, etc.

To do so, simply pass a list of LogRecord attribute names as value of the allowed_reserved_attrs keyword when initializing a GelfFormatter. You can also modify the allowed_reserved_attrs instance variable of an already initialized formatter.

Example
attrs = ["lineno", "module", "filename"]

formatter = GelfFormatter(allowed_reserved_attrs=attrs)
# or
formatter.allowed_reserved_attrs = attrs

logging.debug("starting application...")
{"version": "1.1", "short_message": "starting application...", "timestamp": 1557346554.989846, "level": 6, "host": "my-server", "_lineno": 175, "_module": "myapp", "_filename": "app.py"}

You can optionally customize the name of these additional fields using a logging.Filter (see below).

Context Fields

Having the ability to define a set of additional fields once and have them included in all log messages can be useful to avoid repetitive extra key/value pairs and enable contextual logging.

Python's logging module provides several options to add context to a logger, among which we highlight the logging.LoggerAdapter and logging.Filter.

Between these we recommend a logging.Filter, which is simpler and can be attached directly to a logging.Handler. A logging.Filter can therefore be used locally (on a logging.Logger) or globally (through logging.basicConfig). If you opt for a LoggerAdapter you'll need a logging.Logger to wrap.

You can also use a logging.Filter to reuse/rename any of the reserved logging.LogRecord attributes.

Example
class ContextFilter(logging.Filter):
    def filter(self, record):
        # Add any number of arbitrary additional fields
        record.app = "my-app"
        record.app_version = "1.2.3"
        record.environment = os.environ.get("APP_ENV")

        # Reuse any reserved `logging.LogRecord` attributes
        record.file = record.filename
        record.line = record.lineno
        return True


formatter = GelfFormatter()

handler = logging.StreamHandler(sys.stdout)

handler.setFormatter(formatter)
handler.addFilter(ContextFilter())

logging.basicConfig(level=logging.DEBUG, handlers=[handler])

logging.info("hi", extra=dict(foo="bar"))
{"version": "1.1", "short_message": "hi", "timestamp": 1557431642.189755, "level": 6, "host": "my-server", "_foo": "bar", "_app": "my-app", "_app_version": "1.2.3", "_environment": "development", "_file": "app.py", "_line": 159}

Pretty-Print

Looking for a GELF log pretty-printer? If so, have a look at gelf-pretty :fire:

Contributions

This project adheres to the Contributor Covenant code of conduct. By participating, you are expected to uphold this code. Please refer to our contributing guide for further information.

Project details


Release history Release notifications

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for gelf-formatter, version 0.1.0
Filename, size File type Python version Upload date Hashes
Filename, size gelf_formatter-0.1.0-py2.py3-none-any.whl (7.6 kB) File type Wheel Python version py2.py3 Upload date Hashes View hashes
Filename, size gelf-formatter-0.1.0.tar.gz (8.7 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page