A simple library to emit contextual information in structured logs (JSON)
Project description
Python context-log library
About
context-log is a simple library to emit contextual information in structured logs (JSON).
It works particularly well in a Docker or Serverless (e.g. AWS Lambda) environment where a single thread executes a request and produces a response.
The library uses python threading to store contextual information that is automatically added to all subsequent logs in a contextMap
field.
Because the library uses the Python thread local context it works across packages and modules in a given project.
The approach is loosely based on the Log4j 2 API Thread Context.
Usage
Structured logging can be achieved with the python-json-logger library.
Simply add project dependencies to requirements.txt:
python_json_logger
PyYAML
context-log
Add the code below to the main code module.
Add the following YAML configuration in the resources/logging.yaml
file, which outputs JSON structured logs to stdout
.
version: 1
formatters:
json:
class: .jsonlogger.JsonFormatter
format: '%(asctime)s %(name)s %(levelname)s %(message)s %(filename)s'
handlers:
console:
class: logging.StreamHandler
level: DEBUG
formatter: json
root:
level: DEBUG
handlers:
- console
Use the context_log library to emit logs. Example below.
import logging.config
import yaml
with open('resources/logging.yaml', 'r') as log_config_file:
logging.config.dictConfig(yaml.safe_load(log_config_file))
from context_log import ContextLog
def handler(event, context):
# Clear context (e.g. re-use) and get logger
log = ContextLog.get_logger('handler', True)
log.info('start')
ContextLog.put('ip', '1.2.3.4')
# Helper to add start time in ISO and epoch time
ContextLog.put_request_start_time()
# Process request
sleep(0.1)
# Helper to add end time in ISO and epoch time
# as well as duration in milliseconds
ContextLog.put_request_end_time()
log.info('end')
First log info event:
{
"asctime": "2019-09-19 11:53:20,479",
"name": "handler",
"levelname": "INFO",
"message": "start",
"filename": "test_example.py",
"contextMap": {}
}
Second log info event:
{
"asctime": "2019-09-19 11:53:20,580",
"name": "handler",
"levelname": "INFO",
"message": "end",
"filename": "test_example.py",
"contextMap": {
"ip": "1.2.3.4",
"start-time": "2019-09-19T11:53:20.480085",
"epoch-start-time": 1568890400.480085,
"end-time": "2019-09-19T11:53:20.580513",
"epoch-end-time": 1568890400.580513,
"duration": 100.428}
}
The Detail
The standard logger is wrapped by a LoggerAdapter
. It is therefore imperative that the ContextLog.get_logger(name='<name>', clear=True|False)
call is made to get the logger to emit contextual logs.
Use clear=True
when starting a new request in order to clear the previous context if the thread is re-used. This is typically the case in thread pools and in AWS Lambda's.
To manipulate or retrieve the contextMap
use the following methods:
clear()
put(key, value)
get(key)
get_map()
There are also a number of helpers in an attempt to standardise log output contextMap
fields:
put_request_id(request_id)
put_request_method(request_method)
put_request_path(request_path)
put_response_status(response_status)
put_start_time()
put_end_time()
put_request_user_id(request_user_id)
put_request_client_id(request_client_id)
put_request_primary_ip(primary_ip)
put_request_client_ip(client_ip)
put_request_viewer_country(viewer_country)
put_trigger_source(trigger_source)
Contributing
Pull requests are more than welcome.
Running pytest
Create virtualenv, download dependencies and run tests:
python3 -m venv .venv
source .venv/bin/activate
pip3 install -r tests/requirements.txt
pip3 install -e .
pytest
Running tox
pip3 install --user --upgrade tox
tox
Releasing library to PyPI
Short version from the Packaging Python Projects site.
Install the release tools:
python3 -m pip install --user --upgrade setuptools wheel twine
Remove old distribution(s):
rm -rf dist/
Build the context-log package:
python3 setup.py sdist bdist_wheel
Upload context-log first to Test PyPI:
python3 -m twine upload --repository-url https://test.pypi.org/legacy/ dist/*
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for context_log-0.1.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3b4afbcdad68fb6e756a4189fa2fed2dedd79dc585ac97f1d4b115b0b15d9e74 |
|
MD5 | 2b943b3e56f3373622851f7a6950893f |
|
BLAKE2b-256 | af2ea5f7167defaceda79657cb20f72690b4ac082508db9a2e34c90a4890f464 |