DataHub Prefect Plugin
Automatic lineage and run metadata from Prefect into DataHub — captures flow structure, task inputs/outputs, and run history with minimal setup.
What you can do
- Emit flow and task metadata to DataHub as pipeline runs
- Capture dataset lineage — declare inputs and outputs per task and see them in DataHub
- Configure via Prefect blocks — store your DataHub connection settings as a reusable block
- Works with any DataHub deployment — self-hosted or DataHub Cloud
Installation
pip install prefect-datahub
Quickstart
1. Save your DataHub connection as a Prefect block
from prefect_datahub.datahub_emitter import DatahubEmitter
DatahubEmitter(
datahub_rest_url="http://localhost:8080",
env="PROD",
).save("my-datahub")
2. Use it in your flows
from prefect import flow, task
from prefect_datahub.datahub_emitter import DatahubEmitter
from prefect_datahub.entities import Dataset
emitter = DatahubEmitter.load("my-datahub")
@task
def transform(data, emitter):
emitter.add_task(
inputs=[Dataset("snowflake", "mydb.schema.source_table")],
outputs=[Dataset("snowflake", "mydb.schema.output_table")],
)
return data
@flow
def my_pipeline():
data = extract()
transform(data, emitter)
emitter.emit_flow() # required — emits all metadata at the end
Configuration options
| Option | Default | Description |
|---|---|---|
datahub_rest_url |
http://localhost:8080 |
DataHub GMS URL |
env |
PROD |
Environment tag for assets |
platform_instance |
None |
Platform instance for assets |
token |
None |
Auth token (if GMS auth is enabled) |
Links
1 maintainer
Metadata
1 maintainer
Release files for prefect-datahub 1.7.0.18
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| prefect_datahub-1.7.0.18.tar.gz | 13.8 kB | Without provenance | Details for prefect_datahub-1.7.0.18.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| prefect_datahub-1.7.0.18-py3-none-any.whl | Python 3 | none | any | Details for prefect_datahub-1.7.0.18-py3-none-any.whl |
Total release size: 25.7 kB
Release files / prefect_datahub-1.7.0.18.tar.gz
| Download URL | prefect_datahub-1.7.0.18.tar.gz |
|---|---|
| Size | 13.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
805a7c1b4204093ce0bf7daec61412f2ad1ef35b8f9a5d517c1b5200a0b3ec57
|
|
BLAKE2b-256 checksum How to use checksums |
97bad2fe347769043c063da983d69e01745b486511923254924272975b045346
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.22
|
Release files / prefect_datahub-1.7.0.18-py3-none-any.whl
| Download URL | prefect_datahub-1.7.0.18-py3-none-any.whl |
|---|---|
| Size | 11.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
cb0f49fff20b10e2c849a5e839b060d3ff2688324d6de69453e1a7278e73bf7a
|
|
BLAKE2b-256 checksum How to use checksums |
d06c27b662cab3a90472076a30b9516313b17df697fd46cfe88c589f1e916012
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.22
|
Security for prefect-datahub 1.7.0.18
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
