Parse structured metadata from dbt logs
Project description
dbt log parser
Python package and CLI tool for parsing structured (JSON) data from a dbt log.
Table of Contents
Install
pip install dbt_log_parser
Usage
CLI
$ dbtlp --help
usage: dbtlp [-h] [--log-filepath LOG_FILEPATH] [--outfile OUTFILE]
DBT log parser
optional arguments:
-h, --help show this help message and exit
--log-filepath LOG_FILEPATH
Path to dbt log to parse
--outfile OUTFILE File to write JSON results to
$ dbtlp --log-filepath dbt.log --outfile results.json
API
from dbt_log_parser import parse
report = parse(log_filepath='dbt.log', write_report=False)
Generated Report
Example report structure:
{
"models_found": 469,
"tests_found": 1658,
"snapshots_found": 0,
"analyses_found": 0,
"macros_found": 300,
"operations_found": 0,
"seeds_found": 0,
"sources_found": 204,
"tests_run": 1100,
"tests_runtime_seconds": 97.47,
"tests": [
{
"number": 1,
"name": "accepted_values_dim_order_state_open",
"status": "PASS",
"total_time": 2.11
},
{
"status": "WARN",
"number": 2,
"name": "relationships_fact_contribution_goal_updates_contribution_page_id__contribution_page_id__ref_dim_contribution_page_",
"total_time": 1.73,
"query_results": {
"found": 5,
"expected": 0
},
"query": {
"filepath": "target/compiled/core/schema_test/relationships_fact_contribution_goal_updates_34dae512835158ed459182c173a8c127.sql",
"sql": null,
"file_err": true
}
},
{
"status": "FAIL",
"number": 548326,
"name": "unique_int_home_page_display_resources_you_may_like_event_user_id",
"total_time": 1.16,
"query_results": {
"found": 548326,
"expected": 0
},
"query": {
"filepath": "target/compiled/core/schema_test/unique_int_home_page_display_resources_you_may_like_event_user_id.sql",
"sql": null,
"file_err": true
}
}
]
}
See more in the JSON schema.
Contributing
See CONTRIBUTING.md
Change Log
Unreleased
v0.1.1a0
- Testing GitHub Actions for tag-triggered releases
v0.1.0
- Initial non-alpha release, now that project has improved tests and documentation
v0.1.0a0
- Initial release
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
dbt_log_parser-0.1.1a0.tar.gz
(22.1 kB
view hashes)
Built Distribution
Close
Hashes for dbt_log_parser-0.1.1a0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4eae4c4ed171dfc13e8a75e77d92f4df9164cb94545fe3ecfadbbc3e2f5abed7 |
|
MD5 | fdcb23184fe79fbc61f122eace5664c1 |
|
BLAKE2b-256 | a1718d031e1bcec2d803ce3d2acfdee699736c4038cc97ca020e3821e0c7f57b |