Versatile Data Kit SDK plugin that changes logging output to JSON format.
Project description
This is a POC level implementation of a plugin which changes the logging format of vdk-core to JSON for the purposes of structured data visualization.
The new format has the following fields, separated by tabs:
- timestamp: a timestamp of when the log is made;
- created: the same timestamp in the unix epoch format;
- jobname: the name of the data job;
- level: the logging level - one of INFO, WARNING, DEBUG, ERROR;
- modulename: the name of the module, inside of which the logging call is made;
- filename: the name of the file containing the logging call being made;
- lineno: the number of the line of code, at which the logging call is made;
- funcname: the name of the function, inside which the loggin call is made;
- attemptid: string identifying this particular execution of the data job;
- message: any additional logged information.
The label names follow the labelling recommendations found at http://ltsv.org/. The reason we chose the LTSV naming standard is due to the fact that this plugin was based on a previous LTSV-formatting plugin, as well as the fact that there is no single JSON naming standard.
Additionally, newline characters within the message are escaped.
Usage
Switching vdk logging can be done by simply installing the plugin:
pip install vdk-logging-json
And all logs will be automatically formatted to JSON. They will appear like this:
{"@timestamp":"2021-08-04T12:51:11.532Z","created":"1628070671","jobname":"example-job","level":"DEBUG","modulename":"vdk.internal.trino_connection","filename":"managed_connection_base.py","lineno":"69","funcname":"connect","attemptid":"1628070671-452613-739749","message":"Established <trino.dbapi.Connection object at 0x10b9b1d30>"}
{"@timestamp":"2021-08-04T12:51:11.532Z","created:1628070671","jobname":"example-job","level":"DEBUG","modulename":"vdk.internal.trino_connection","filename":"managed_cursor.py","lineno":"29","funcname":"execute","attemptid":"1628070671-452613-739749","message":"Executing query: select 1"}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Hashes for vdk-logging-json-0.1.376671772.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 378ba8d97b3ed355c87b4a687301783aafbefade36cde8ed5637bf88d48abbb3 |
|
MD5 | fed5486991c6f72be53223b7843e9b1a |
|
BLAKE2b-256 | 2aae57eace7b2e193f4e3bcc211aa3c8e93782d3c70f5761a4460197065da88d |