Pre-release
This release is a pre-release and may not be stable for production use.
DataHub Prefect Plugin
Automatic lineage and run metadata from Prefect into DataHub — captures flow structure, task inputs/outputs, and run history with minimal setup.
What you can do
- Emit flow and task metadata to DataHub as pipeline runs
- Capture dataset lineage — declare inputs and outputs per task and see them in DataHub
- Configure via Prefect blocks — store your DataHub connection settings as a reusable block
- Works with any DataHub deployment — self-hosted or DataHub Cloud
Installation
pip install prefect-datahub
Quickstart
1. Save your DataHub connection as a Prefect block
from prefect_datahub.datahub_emitter import DatahubEmitter
DatahubEmitter(
datahub_rest_url="http://localhost:8080",
env="PROD",
).save("my-datahub")
2. Use it in your flows
from prefect import flow, task
from prefect_datahub.datahub_emitter import DatahubEmitter
from prefect_datahub.entities import Dataset
emitter = DatahubEmitter.load("my-datahub")
@task
def transform(data, emitter):
emitter.add_task(
inputs=[Dataset("snowflake", "mydb.schema.source_table")],
outputs=[Dataset("snowflake", "mydb.schema.output_table")],
)
return data
@flow
def my_pipeline():
data = extract()
transform(data, emitter)
emitter.emit_flow() # required — emits all metadata at the end
Configuration options
| Option | Default | Description |
|---|---|---|
datahub_rest_url |
http://localhost:8080 |
DataHub GMS URL |
env |
PROD |
Environment tag for assets |
platform_instance |
None |
Platform instance for assets |
token |
None |
Auth token (if GMS auth is enabled) |
Links
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file prefect_datahub-1.6.0.17rc2.tar.gz.
File metadata
- Download URL: prefect_datahub-1.6.0.17rc2.tar.gz
- Upload date:
- Size: 13.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.10.20
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
ec381356d18718e6611e23be53e078b9ec373feab68a48932dcfcc905e6f60b3
|
|
| MD5 |
cbfb0931ce596fd2466fc94ca5fcf55e
|
|
| BLAKE2b-256 |
40948fb0d0607522859291f1853455d4d0aced675f6b41291d24bbb6bd0aba42
|
File details
Details for the file prefect_datahub-1.6.0.17rc2-py3-none-any.whl.
File metadata
- Download URL: prefect_datahub-1.6.0.17rc2-py3-none-any.whl
- Upload date:
- Size: 11.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.10.20
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
571139e733b2762450a4abdc7e9864765601b0d7d2a53e52422ffa791e900315
|
|
| MD5 |
6d558ad3dabb9e927f1f3f6e476d1f98
|
|
| BLAKE2b-256 |
f5d7a72737e4404f8ffcfdea09e0029ab25721c2e95d7ee29ac64e5f218c4e51
|
