OpenLineage integration with dbt
Project description
OpenLineage dbt integration
Wrapper script for automatic metadata collection from dbt
Features
Metadata
- Model run lifecycle
- Model inputs / outputs
Requirements
Right now, openlineage-dbt
only supports bigquery
, snowflake
, spark
and redshift
dbt adapters.
Installation
$ pip3 install openlineage-dbt
To install from source, run:
$ pip install .
Configuration
HTTP
Backend Environment Variables
openlineage-dbt
uses the OpenLineage client to push data to the OpenLineage backend.
The OpenLineage client depends on environment variables:
OPENLINEAGE_URL
- point to service which will consume OpenLineage eventsOPENLINEAGE_API_KEY
- set if consumer of OpenLineage events requiresBearer
authentication keyOPENLINEAGE_NAMESPACE
- set if you are using something other than thedefault
namespace for job namespace.
Usage
To begin collecting dbt metadata with OpenLineage, replace dbt run
with dbt-ol run
.
Additional table and column level metadata will be available if catalog.json
, a result of running dbt docs generate
, will be found in the target directory.
SPDX-License-Identifier: Apache-2.0
Copyright 2018-2023 contributors to the OpenLineage project
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for openlineage_dbt-0.20.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 07dcfa0b19a8c03d50a3e1d875c29826ff2f4d880f551e0443af43549c6e8a3f |
|
MD5 | 5c2a2a1c569b80cf0081c809746f0c6e |
|
BLAKE2b-256 | d0d5f076ecd9f0111871fea6d93403a924c1b7699ad73d4bf012eb29b1d9187c |