A simple library to emit contextual information in structured logs (JSON)
Project description
Python context-log library
About
context-log is a simple library to emit contextual information in structured logs (JSON).
It works particularly well in a Docker or Serverless (e.g. AWS Lambda) environment where a single thread executes a request and produces a response.
The library uses python threading to store contextual information that is automatically added to all subsequent logs in a contextMap
field.
Because the library uses the Python thread local context it works across packages and modules in a given project.
The approach is loosely based on the Log4j 2 API Thread Context.
Usage
Structured logging can be achieved with the python-json-logger library.
Simply add project dependencies to requirements.txt:
python_json_logger
PyYAML
context-log
Add the code below to the main code module.
Add the following YAML configuration in the resources/logging.yaml
file, which outputs JSON structured logs to stdout
.
version: 1
formatters:
json:
class: .jsonlogger.JsonFormatter
format: '%(asctime)s %(name)s %(levelname)s %(message)s %(filename)s'
handlers:
console:
class: logging.StreamHandler
level: DEBUG
formatter: json
root:
level: DEBUG
handlers:
- console
Use the context_log library to emit logs. Example below.
import logging.config
import yaml
with open('resources/logging.yaml', 'r') as log_config_file:
logging.config.dictConfig(yaml.safe_load(log_config_file))
from context_log import ContextLog
def handler(event, context):
# Clear context (e.g. re-use) and get logger
log = ContextLog.get_logger('handler', True)
log.info('start')
ContextLog.put('ip', '1.2.3.4')
# Helper to add start time in ISO and epoch time
ContextLog.put_request_start_time()
# Process request
sleep(0.1)
# Helper to add end time in ISO and epoch time
# as well as duration in milliseconds
ContextLog.put_request_end_time()
log.info('end')
First log info event:
{
"asctime": "2019-09-19 11:53:20,479",
"name": "handler",
"levelname": "INFO",
"message": "start",
"filename": "test_example.py",
"contextMap": {}
}
Second log info event:
{
"asctime": "2019-09-19 11:53:20,580",
"name": "handler",
"levelname": "INFO",
"message": "end",
"filename": "test_example.py",
"contextMap": {
"ip": "1.2.3.4",
"start-time": "2019-09-19T11:53:20.480085",
"epoch-start-time": 1568890400.480085,
"end-time": "2019-09-19T11:53:20.580513",
"epoch-end-time": 1568890400.580513,
"duration": 100.428}
}
The Detail
The standard logger is wrapped by a LoggerAdapter
. It is therefore imperative that the ContextLog.get_logger(name='<name>', clear=True|False)
call is made to get the logger to emit contextual logs.
Use clear=True
when starting a new request in order to clear the previous context if the thread is re-used. This is typically the case in thread pools and in AWS Lambda's.
To manipulate or retrieve the contextMap
use the following methods:
clear()
put(key, value)
get(key)
get_map()
There are also a number of helpers in an attempt to standardise log output contextMap
fields:
put_request_id(request_id)
put_request_method(request_method)
put_request_path(request_path)
put_response_status(response_status)
put_start_time()
put_end_time()
put_request_user_id(request_user_id)
put_request_client_id(request_client_id)
put_request_primary_ip(primary_ip)
put_request_client_ip(client_ip)
put_request_viewer_country(viewer_country)
put_trigger_source(trigger_source)
Contributing
Pull requests are more than welcome.
Running pytest
Create virtualenv, download dependencies and run tests:
python3 -m venv .venv
source .venv/bin/activate
pip3 install -r tests/requirements.txt
pip3 install -e .
pytest
Running tox
pip3 install --user --upgrade tox
tox
Releasing library to PyPI
Short version from the Packaging Python Projects site.
Install the release tools:
python3 -m pip install --user --upgrade setuptools wheel twine
Remove old distribution(s):
rm -rf dist/
Build the context-log package:
python3 setup.py sdist bdist_wheel
Upload context-log first to Test PyPI:
python3 -m twine upload --repository-url https://test.pypi.org/legacy/ dist/*
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file context-log-0.1.2.tar.gz
.
File metadata
- Download URL: context-log-0.1.2.tar.gz
- Upload date:
- Size: 4.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.4.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.7.4
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | c4022a86ebe2c55cb456ade01db05f696177437c7f5dc1f9cb8c05240585082a |
|
MD5 | 9433d02de03e11be22a4daf3cba91e4c |
|
BLAKE2b-256 | e797de326181cb1238675abb50ce34b6ec17474f1416f39eb1f0a08a17f4d304 |
File details
Details for the file context_log-0.1.2-py3-none-any.whl
.
File metadata
- Download URL: context_log-0.1.2-py3-none-any.whl
- Upload date:
- Size: 6.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.4.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.7.4
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3b4afbcdad68fb6e756a4189fa2fed2dedd79dc585ac97f1d4b115b0b15d9e74 |
|
MD5 | 2b943b3e56f3373622851f7a6950893f |
|
BLAKE2b-256 | af2ea5f7167defaceda79657cb20f72690b4ac082508db9a2e34c90a4890f464 |