Skip to main content

A simple library to emit contextual information in structured logs (JSON)

Project description

Python context-log library

About

context-log is a simple library to emit contextual information in structured logs (JSON).

It works particularly well in a Docker or Serverless (e.g. AWS Lambda) environment where a single thread executes a request and produces a response.

The library uses python threading to store contextual information that is automatically added to all subsequent logs in a contextMap field.

Because the library uses the Python thread local context it works across packages and modules in a given project.

The approach is loosely based on the Log4j 2 API Thread Context.

Usage

Structured logging can be achieved with the python-json-logger library.

Simply add project dependencies to requirements.txt:

python_json_logger
PyYAML
context-log

Add the code below to the main code module.

Add the following YAML configuration in the resources/logging.yaml file, which outputs JSON structured logs to stdout.

version: 1
formatters:
  json:
    class:  .jsonlogger.JsonFormatter
    format: '%(asctime)s %(name)s %(levelname)s %(message)s %(filename)s'
handlers:
  console:
    class: logging.StreamHandler
    level: DEBUG
    formatter: json
root:
  level: DEBUG
  handlers:
    - console

Use the context_log library to emit logs. Example below.

import logging.config
import yaml

with open('resources/logging.yaml', 'r') as log_config_file:
  logging.config.dictConfig(yaml.safe_load(log_config_file))

from context_log import ContextLog

def handler(event, context):
    # Clear context (e.g. re-use) and get logger
    log = ContextLog.get_logger('handler', True)
    log.info('start')

    ContextLog.put('ip', '1.2.3.4')

    # Helper to add start time in ISO and epoch time
    ContextLog.put_request_start_time()

    # Process request
    sleep(0.1)

    # Helper to add end time in ISO and epoch time
    # as well as duration in milliseconds
    ContextLog.put_request_end_time()

    log.info('end')

First log info event:

{
    "asctime": "2019-09-19 11:53:20,479",
    "name": "handler",
    "levelname": "INFO",
    "message": "start",
    "filename": "test_example.py",
    "contextMap": {}
}

Second log info event:

{
  "asctime": "2019-09-19 11:53:20,580",
  "name": "handler",
  "levelname": "INFO",
  "message": "end",
  "filename": "test_example.py",
  "contextMap": {
      "ip": "1.2.3.4",
      "start-time": "2019-09-19T11:53:20.480085",
      "epoch-start-time": 1568890400.480085,
      "end-time": "2019-09-19T11:53:20.580513",
      "epoch-end-time": 1568890400.580513,
      "duration": 100.428}
}

The Detail

The standard logger is wrapped by a LoggerAdapter. It is therefore imperative that the ContextLog.get_logger(name='<name>', clear=True|False) call is made to get the logger to emit contextual logs.

Use clear=True when starting a new request in order to clear the previous context if the thread is re-used. This is typically the case in thread pools and in AWS Lambda's.

To manipulate or retrieve the contextMap use the following methods:

  • clear()
  • put(key, value)
  • get(key)
  • get_map()

There are also a number of helpers in an attempt to standardise log output contextMap fields:

  • put_request_id(request_id)
  • put_request_method(request_method)
  • put_request_path(request_path)
  • put_response_status(response_status)
  • put_start_time()
  • put_end_time()
  • put_request_user_id(request_user_id)
  • put_request_client_id(request_client_id)
  • put_request_primary_ip(primary_ip)
  • put_request_client_ip(client_ip)
  • put_request_viewer_country(viewer_country)
  • put_trigger_source(trigger_source)

Contributing

Pull requests are more than welcome.

Running pytest

Create virtualenv, download dependencies and run tests:

python3 -m venv .venv
source .venv/bin/activate
pip3 install -r tests/requirements.txt
pip3 install -e .
pytest

Running tox

pip3 install --user --upgrade tox
tox

Releasing library to PyPI

Short version from the Packaging Python Projects site.

Install the release tools:

python3 -m pip install --user --upgrade setuptools wheel twine

Remove old distribution(s):

rm -rf dist/

Build the context-log package:

python3 setup.py sdist bdist_wheel

Upload context-log first to Test PyPI:

python3 -m twine upload --repository-url https://test.pypi.org/legacy/ dist/*

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

context-log-0.1.2.tar.gz (4.8 kB view details)

Uploaded Source

Built Distribution

context_log-0.1.2-py3-none-any.whl (6.0 kB view details)

Uploaded Python 3

File details

Details for the file context-log-0.1.2.tar.gz.

File metadata

  • Download URL: context-log-0.1.2.tar.gz
  • Upload date:
  • Size: 4.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.4.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.7.4

File hashes

Hashes for context-log-0.1.2.tar.gz
Algorithm Hash digest
SHA256 c4022a86ebe2c55cb456ade01db05f696177437c7f5dc1f9cb8c05240585082a
MD5 9433d02de03e11be22a4daf3cba91e4c
BLAKE2b-256 e797de326181cb1238675abb50ce34b6ec17474f1416f39eb1f0a08a17f4d304

See more details on using hashes here.

File details

Details for the file context_log-0.1.2-py3-none-any.whl.

File metadata

  • Download URL: context_log-0.1.2-py3-none-any.whl
  • Upload date:
  • Size: 6.0 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.4.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.7.4

File hashes

Hashes for context_log-0.1.2-py3-none-any.whl
Algorithm Hash digest
SHA256 3b4afbcdad68fb6e756a4189fa2fed2dedd79dc585ac97f1d4b115b0b15d9e74
MD5 2b943b3e56f3373622851f7a6950893f
BLAKE2b-256 af2ea5f7167defaceda79657cb20f72690b4ac082508db9a2e34c90a4890f464

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page