Versatile Data Kit SDK plugin that changes logging output to JSON format.
Project description
This is a POC level implementation of a plugin which changes the logging format of vdk-core to JSON for the purposes of structured data visualization.
The new format has the following fields, separated by tabs:
- timestamp: a timestamp of when the log is made;
- created: the same timestamp in the unix epoch format;
- jobname: the name of the data job;
- level: the logging level - one of INFO, WARNING, DEBUG, ERROR;
- modulename: the name of the module, inside of which the logging call is made;
- filename: the name of the file containing the logging call being made;
- lineno: the number of the line of code, at which the logging call is made;
- funcname: the name of the function, inside which the loggin call is made;
- attemptid: string identifying this particular execution of the data job;
- message: any additional logged information.
The label names follow the labelling recommendations found at http://ltsv.org/. The reason we chose the LTSV naming standard is due to the fact that this plugin was based on a previous LTSV-formatting plugin, as well as the fact that there is no single JSON naming standard.
Usage
Switching vdk logging can be done by simply installing the plugin:
pip install vdk-logging-json
And all logs will be automatically formatted to JSON. They will appear like this:
{"@timestamp":"2021-08-04T12:51:11.532Z","created":"1628070671","jobname":"example-job","level":"DEBUG","modulename":"taurus.vdk.trino_connection","filename":"managed_connection_base.py","lineno":"69","funcname":"connect","attemptid":"1628070671-452613-739749","message":"Established <trino.dbapi.Connection object at 0x10b9b1d30>"}
{"@timestamp":"2021-08-04T12:51:11.532Z","created:1628070671","jobname":"example-job","level":"DEBUG","modulename":"taurus.vdk.trino_connection","filename":"managed_cursor.py","lineno":"29","funcname":"execute","attemptid":"1628070671-452613-739749","message":"Executing query: select 1"}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Hashes for vdk-logging-json-0.1.352665786.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 10d899f75d4a19cb71b2be0528557bed9d21eefd104324c535563ece2eb6055b |
|
MD5 | ab1b6af53b61032bcc151ca117532f32 |
|
BLAKE2b-256 | b9917f6ab106d9da282bd90280d7cd3f80d21bbbe8bc220c13f4ab5ddcc05a52 |