Skip to main content

The Marquez BigQuery adapter plugin for dbt (data build tool)

Project description

dbt-bigquery-openlineage

Plugin for dbt (data build tool) for automatic metadata and lineage collection.

This integration is experimental, and there is high possibility that it will change in the future, as together with dbt developers we're figuring what's the best way to extract metadata from it.

Requirements

Python 3.6+
sqlparse>=0.3.1

Installation

$ pip install marquez-dbt-bigquery

Configuration

You need to configure dbt's profiles.yaml file. This plugin introduces additional type: openlineage_bigquery.

It handles the same configuration flags as original bigquery plugin, with the addition of OpenLineage related ones:

    openlineage_url - sets URL of OpenLineage server where events will be pushed
    openlineage_timeout - optional - sets timeout for connecting to OpenLineage server. By default it's set to 5 seconds.
    openlineage_api_key - optional - sets api key for OpenLineage server

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

marquez-dbt-bigquery-0.16.1.tar.gz (5.9 kB view hashes)

Uploaded source

Built Distribution

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page