Skip to main content

OpenLineage integration with dbt

Project description

OpenLineage dbt integration

Wrapper script for dbt for automatic metadata collection

Features

Metadata

  • Model run lifecycle
  • Model inputs / outputs

Requirements

Right now, openlineage-dbt only supports bigquery, snowflake, spark and redshift dbt adapters.

Installation

$ pip3 install openlineage-dbt

To install from source, run:

$ pip install .

Configuration

HTTP Backend Environment Variables

openlineage-dbt uses OpenLineage client to push data to OpenLineage backend.

OpenLineage client depends on environment variables:

  • OPENLINEAGE_URL - point to service which will consume OpenLineage events
  • OPENLINEAGE_API_KEY - set if consumer of OpenLineage events requires Bearer authentication key
  • OPENLINEAGE_NAMESPACE - set if you are using something other than the default namespace for job namespace.

Usage

To begin collecting dbt metadata with OpenLineage, replace dbt run with dbt-ol run.

Additional table and column level metadata will be available if catalog.json, result of running dbt docs generate will be found in target directory.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

openlineage-dbt-0.10.0.tar.gz (4.0 kB view hashes)

Uploaded source

Built Distribution

openlineage_dbt-0.10.0-py3-none-any.whl (4.0 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page