DataHub Airflow Plugin
Automatic lineage and run metadata from Apache Airflow into DataHub — captures DAG structure, task inputs/outputs, and run history with no manual instrumentation.
What you can do
- Capture pipeline lineage — automatically extract dataset-level and column-level lineage from SQL operators
- Track run history — record task execution status, duration, and failures in DataHub
- Enhance OpenLineage — patches Airflow's OpenLineage extractors with DataHub's advanced SQL parser for richer lineage
- Support multiple emitters — send metadata via REST, Kafka, or file
Version compatibility
| Airflow Version | Support |
|---|---|
| 3.0+ | ✅ Fully supported |
| 2.x | ❌ Use acryl-datahub-airflow-plugin <= 1.6.0 |
Installation
pip install acryl-datahub-airflow-plugin
# With Kafka emitter
pip install 'acryl-datahub-airflow-plugin[datahub-kafka]'
Configuration
Add to airflow.cfg:
[datahub]
enabled = True
conn_id = datahub_rest_default # Airflow connection pointing to your DataHub GMS
Set up the Airflow connection:
airflow connections add datahub_rest_default \
--conn-type HTTP \
--conn-host http://localhost:8080
The plugin activates automatically — no changes to your DAG code required.
Key configuration options
| Option | Default | Description |
|---|---|---|
enable_extractors |
True |
Enhance OpenLineage extractors |
patch_sql_parser |
True |
Use DataHub's SQL parser for column-level lineage |
enable_multi_statement_sql_parsing |
False |
Resolve temp tables across multi-statement tasks |
Links
2 maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file acryl_datahub_airflow_plugin-1.7.0.4.tar.gz.
File metadata
- Download URL: acryl_datahub_airflow_plugin-1.7.0.4.tar.gz
- Upload date:
- Size: 58.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.10.20
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
996fb5c0456d36951c4136edcd1f38d0f890f75e5a0de29dff57895e01a16d48
|
|
| MD5 |
04380838b775cad28a16c26cea583511
|
|
| BLAKE2b-256 |
728398b40cc56b099e0c39904c5b64822b5d68572339693ef6dbea50cfd81590
|
File details
Details for the file acryl_datahub_airflow_plugin-1.7.0.4-py3-none-any.whl.
File metadata
- Download URL: acryl_datahub_airflow_plugin-1.7.0.4-py3-none-any.whl
- Upload date:
- Size: 75.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.10.20
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
6894b4e598209c4a532017a6bdd09471e576d60757eadd95170aaacc233a75f9
|
|
| MD5 |
d4fde959044571e0e97feeed40c87440
|
|
| BLAKE2b-256 |
20c24874fc77f5ca5b5acca393a11474317fd03bff61c522fc64a9f061bab467
|
