Skip to main content

DataHub Prefect Plugin

Automatic lineage and run metadata from Prefect into DataHub — captures flow structure, task inputs/outputs, and run history with minimal setup.

What you can do

  • Emit flow and task metadata to DataHub as pipeline runs
  • Capture dataset lineage — declare inputs and outputs per task and see them in DataHub
  • Configure via Prefect blocks — store your DataHub connection settings as a reusable block
  • Works with any DataHub deployment — self-hosted or DataHub Cloud

Installation

pip install prefect-datahub

Quickstart

1. Save your DataHub connection as a Prefect block

from prefect_datahub.datahub_emitter import DatahubEmitter

DatahubEmitter(
    datahub_rest_url="http://localhost:8080",
    env="PROD",
).save("my-datahub")

2. Use it in your flows

from prefect import flow, task
from prefect_datahub.datahub_emitter import DatahubEmitter
from prefect_datahub.entities import Dataset

emitter = DatahubEmitter.load("my-datahub")

@task
def transform(data, emitter):
    emitter.add_task(
        inputs=[Dataset("snowflake", "mydb.schema.source_table")],
        outputs=[Dataset("snowflake", "mydb.schema.output_table")],
    )
    return data

@flow
def my_pipeline():
    data = extract()
    transform(data, emitter)
    emitter.emit_flow()   # required — emits all metadata at the end

Configuration options

Option Default Description
datahub_rest_url http://localhost:8080 DataHub GMS URL
env PROD Environment tag for assets
platform_instance None Platform instance for assets
token None Auth token (if GMS auth is enabled)

Metadata

Release files for prefect-datahub 1.7.0.20

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for prefect-datahub 1.7.0.20
File Size Uploaded Provenance Actions
prefect_datahub-1.7.0.20.tar.gz 13.8 kB Without provenance Details for prefect_datahub-1.7.0.20.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for prefect-datahub 1.7.0.20
File Interpreter ABI Platform Actions
prefect_datahub-1.7.0.20-py3-none-any.whl Python 3 none any Details for prefect_datahub-1.7.0.20-py3-none-any.whl

Total release size: 25.7 kB

Release files / prefect_datahub-1.7.0.20.tar.gz

Download URL prefect_datahub-1.7.0.20.tar.gz
Size 13.8 kB
Tags Source
SHA-256 checksum
How to use checksums
8c4ad7f59ed1e7fb1a0eb1085d71d4e88d5acc04a8e3d5229ab70c5ce08b6dfc
BLAKE2b-256 checksum
How to use checksums
a9ca7258415c1b94e65557b2678cff9f6b77512a974be0b53d56a010f8bda0fe
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.22

Release files / prefect_datahub-1.7.0.20-py3-none-any.whl

Download URL prefect_datahub-1.7.0.20-py3-none-any.whl
Size 11.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
ba7c3f2b6ca87b73c5537f86e5f94eedd53c1dee8670a6535374690bafd55d79
BLAKE2b-256 checksum
How to use checksums
ceae85a061698d73d81c97b9881b62165485223eb191f0864f22cc7419ce35d7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.22

Security for prefect-datahub 1.7.0.20

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.7.0.20 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page