This release is a pre-release and may not be stable for production use.
DataHub Airflow Plugin
Automatic lineage and run metadata from Apache Airflow into DataHub — captures DAG structure, task inputs/outputs, and run history with no manual instrumentation.
What you can do
- Capture pipeline lineage — automatically extract dataset-level and column-level lineage from SQL operators
- Track run history — record task execution status, duration, and failures in DataHub
- Enhance OpenLineage — patches Airflow's OpenLineage extractors with DataHub's advanced SQL parser for richer lineage
- Support multiple emitters — send metadata via REST, Kafka, or file
Version compatibility
| Airflow Version | Support |
|---|---|
| 3.0+ | ✅ Fully supported |
| 2.x | ❌ Use acryl-datahub-airflow-plugin <= 1.6.0 |
Installation
pip install acryl-datahub-airflow-plugin
# With Kafka emitter
pip install 'acryl-datahub-airflow-plugin[datahub-kafka]'
Configuration
Add to airflow.cfg:
[datahub]
enabled = True
conn_id = datahub_rest_default # Airflow connection pointing to your DataHub GMS
Set up the Airflow connection:
airflow connections add datahub_rest_default \
--conn-type HTTP \
--conn-host http://localhost:8080
The plugin activates automatically — no changes to your DAG code required.
Key configuration options
| Option | Default | Description |
|---|---|---|
enable_extractors |
True |
Enhance OpenLineage extractors |
patch_sql_parser |
True |
Use DataHub's SQL parser for column-level lineage |
enable_multi_statement_sql_parsing |
False |
Resolve temp tables across multi-statement tasks |
Links
2 maintainers
Metadata
2 maintainers
Release files for acryl-datahub-airflow-plugin 1.7.0.15rc6
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| acryl_datahub_airflow_plugin-1.7.0.15rc6.tar.gz | 61.6 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| acryl_datahub_airflow_plugin-1.7.0.15rc6-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 141.7 kB
Release files / acryl_datahub_airflow_plugin-1.7.0.15rc6.tar.gz
| Download URL | acryl_datahub_airflow_plugin-1.7.0.15rc6.tar.gz |
|---|---|
| Size | 61.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b86e1dba2f8b2165d714d86566abe6d0860ac2665922527b3f15ba9a379ad5fe
|
|
BLAKE2b-256 checksum How to use checksums |
66cd32d4961e89e59dce555eba4f10b1e2fa0edd31f0e4f80da460680b20ac4e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.22
|
Release files / acryl_datahub_airflow_plugin-1.7.0.15rc6-py3-none-any.whl
| Download URL | acryl_datahub_airflow_plugin-1.7.0.15rc6-py3-none-any.whl |
|---|---|
| Size | 80.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e6fe18042e6cb445b371fb85e062e07543e2e57874df48a385158702b1d4d66b
|
|
BLAKE2b-256 checksum How to use checksums |
6c9f83c18c88bbe45dc1aa50bd2b176e0e898f5e3cebae5c0b0ba6ef5a701197
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.22
|
Security for acryl-datahub-airflow-plugin 1.7.0.15rc6
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
