Skip to main content

DataHub Prefect Plugin

Automatic lineage and run metadata from Prefect into DataHub — captures flow structure, task inputs/outputs, and run history with minimal setup.

What you can do

  • Emit flow and task metadata to DataHub as pipeline runs
  • Capture dataset lineage — declare inputs and outputs per task and see them in DataHub
  • Configure via Prefect blocks — store your DataHub connection settings as a reusable block
  • Works with any DataHub deployment — self-hosted or DataHub Cloud

Installation

pip install prefect-datahub

Quickstart

1. Save your DataHub connection as a Prefect block

from prefect_datahub.datahub_emitter import DatahubEmitter

DatahubEmitter(
    datahub_rest_url="http://localhost:8080",
    env="PROD",
).save("my-datahub")

2. Use it in your flows

from prefect import flow, task
from prefect_datahub.datahub_emitter import DatahubEmitter
from prefect_datahub.entities import Dataset

emitter = DatahubEmitter.load("my-datahub")

@task
def transform(data, emitter):
    emitter.add_task(
        inputs=[Dataset("snowflake", "mydb.schema.source_table")],
        outputs=[Dataset("snowflake", "mydb.schema.output_table")],
    )
    return data

@flow
def my_pipeline():
    data = extract()
    transform(data, emitter)
    emitter.emit_flow()   # required — emits all metadata at the end

Configuration options

Option Default Description
datahub_rest_url http://localhost:8080 DataHub GMS URL
env PROD Environment tag for assets
platform_instance None Platform instance for assets
token None Auth token (if GMS auth is enabled)

Metadata

Release files for prefect-datahub 1.8.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for prefect-datahub 1.8.0
File Size Uploaded Provenance Actions
prefect_datahub-1.8.0.tar.gz 13.8 kB Without provenance Details for prefect_datahub-1.8.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for prefect-datahub 1.8.0
File Interpreter ABI Platform Actions
prefect_datahub-1.8.0-py3-none-any.whl Python 3 none any Details for prefect_datahub-1.8.0-py3-none-any.whl

Total release size: 25.7 kB

Release files / prefect_datahub-1.8.0.tar.gz

Download URL prefect_datahub-1.8.0.tar.gz
Size 13.8 kB
Tags Source
SHA-256 checksum
How to use checksums
d2fdcc234989c7946ba81d569adc410c93616919e6908e0aadf597387b1de66b
BLAKE2b-256 checksum
How to use checksums
e40eb181d320a93baf293d1e0702f1c2551398ac8cd50e645fd62e5fa5d430dd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.22

Release files / prefect_datahub-1.8.0-py3-none-any.whl

Download URL prefect_datahub-1.8.0-py3-none-any.whl
Size 11.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b3f38ff7f33cbb2300c3a9bc8235a3cdf38355c24da56859fcc16d872c381cf9
BLAKE2b-256 checksum
How to use checksums
307cb2f76d3c2ff0b56afb1a8e33965e8e1dbc645a1f7f3478d63be0bb95f8ae
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.22

Security for prefect-datahub 1.8.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.8.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page