Skip to main content

OpenLineage integration with dbt

Project description

OpenLineage dbt integration

Wrapper script for automatic metadata collection from dbt

Features

Metadata

  • Model run lifecycle
  • Model inputs / outputs

Requirements

Right now, openlineage-dbt only supports bigquery, snowflake, spark and redshift dbt adapters.

Installation

$ pip3 install openlineage-dbt

To install from source, run:

$ pip install .

Configuration

HTTP Backend Environment Variables

openlineage-dbt uses the OpenLineage client to push data to the OpenLineage backend.

The OpenLineage client depends on environment variables:

  • OPENLINEAGE_URL - point to service which will consume OpenLineage events
  • OPENLINEAGE_API_KEY - set if consumer of OpenLineage events requires Bearer authentication key
  • OPENLINEAGE_NAMESPACE - set if you are using something other than the default namespace for job namespace.

Logging

In addition to conventional logging approaches, the OpenLineage dbt wrapper script provides an alternative way of configuring its logging behavior. By setting the OPENLINEAGE_DBT_LOGGING environment variable, you can establish the logging level for the openlineage.dbt and its child modules.

You can also set log level of dbtol which is deprecated.

Usage

To begin collecting dbt metadata with OpenLineage, replace dbt run with dbt-ol run.

Additional table and column level metadata will be available if catalog.json, a result of running dbt docs generate, will be found in the target directory.

If you need to send events without running the job you can use the command dbt-ol send-events, it will send the metadata of your last run without running the job.


SPDX-License-Identifier: Apache-2.0
Copyright 2018-2024 contributors to the OpenLineage project

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

openlineage_dbt-1.25.0.tar.gz (4.9 kB view details)

Uploaded Source

Built Distribution

openlineage_dbt-1.25.0-py3-none-any.whl (4.7 kB view details)

Uploaded Python 3

File details

Details for the file openlineage_dbt-1.25.0.tar.gz.

File metadata

  • Download URL: openlineage_dbt-1.25.0.tar.gz
  • Upload date:
  • Size: 4.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.8.20

File hashes

Hashes for openlineage_dbt-1.25.0.tar.gz
Algorithm Hash digest
SHA256 c1b4be966a5ee49f5e79509b25c83e2bd91a74c43f97ea1f2d9cbe6ce5c14f57
MD5 66359e42ef23c3d8b1c06c4d6c04e95d
BLAKE2b-256 58162d75df2692347fd12a8c3308086679e432d1dcfa4e5654a18f6c54a88fb7

See more details on using hashes here.

File details

Details for the file openlineage_dbt-1.25.0-py3-none-any.whl.

File metadata

File hashes

Hashes for openlineage_dbt-1.25.0-py3-none-any.whl
Algorithm Hash digest
SHA256 ca1989bdcddc3b2a2390c6acbd864e778fa071d00838aba2d3253a683165ce37
MD5 5e4651ad45b45a34128d6e756a444d1d
BLAKE2b-256 1d6aec08a16ca194636e49b02b7504a1d93cf0e6af5f122472bca379bb02a479

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page