airflow-lineage-toolkit
Answer "what breaks if I change this DAG?" without running Airflow.
airflow-lineage-toolkit reads your Airflow DAG source files with Python's ast
module and builds a complete dependency graph: every task, every >>, every
cross-DAG sensor and trigger, plus what each task actually runs (Databricks main
class / JAR / notebook / wheel, dbt Cloud job id).
No Airflow instance. No metadata database. No importing your DAG code. Just the standard library pointed at a folder.
pip install airflow-lineage-toolkit
airflow-lineage extract --dag-dir dags --var-dir variables
airflow-lineage query impact orders_ingestion_daily
airflow-lineage draw orders_ingestion_daily --open
Why
Most lineage tooling needs a live scheduler, a plugin, or a running task to emit events. That is useless when the question is asked before the change lands:
- Which DAGs break if I retire this one?
- What is the full upstream chain behind this table?
- Which tasks still point at that legacy JAR?
- Is my hand-drawn architecture diagram still true?
Static analysis answers all of those from source, in seconds, in CI, on a laptop, on a branch that has never been deployed.
What it does
| Intra-DAG lineage | >>, <<, set_upstream/downstream, chain(), list fan-out/fan-in |
| Cross-DAG lineage | ExternalTaskSensor, TriggerDagRunOperator, custom sensor classes, and producer/consumer edges inferred from Variable.set/Variable.get execution-time markers |
| Dynamic tasks | tasks built in for loops, comprehensions, factory functions, range/enumerate/zip, f-string and .format() task ids |
| Config resolution | reads a directory of exported Airflow Variable JSON files, so Variable.get("cfg", deserialize_json=True)["tables"] resolves to real values |
| Conditional branches | evaluates if/elif/else statically and captures only the branch that is actually active |
| Task actions | Databricks main_class, JAR, notebook path, python file, wheel package/entry point, parameters; dbt Cloud job_id |
| Integrity audit | fails loudly on phantom edges, disconnected dynamic tasks, self-loops, unresolved template ids |
Truthfulness over completeness
A value that cannot be resolved statically is left null and surfaced under
unresolved_references — never guessed. Shape guards reject anything that looks
like leaked source text masquerading as a resolved value (main_class must be a
valid dotted class name; paths must not contain brackets or quotes). A wrong
"resolved" value is worse than an honest gap, especially when the output feeds a
migration inventory.
Install
pip install airflow-lineage-toolkit # core: standard library only
pip install "airflow-lineage-toolkit[ui]" # + local web UI (FastAPI)
pip install "airflow-lineage-toolkit[excel]" # + .xlsx export (openpyxl)
Python 3.9+. The core extractor, query CLI, HTML visualizer, auditor and CSV/JSON export have zero runtime dependencies — they run in any CI image.
Quickstart
The repository ships a small synthetic Airflow project under examples/ that
exercises every hard pattern, so you can see real output immediately:
airflow-lineage extract --dag-dir examples/dags --var-dir examples/variables
airflow-lineage audit
airflow-lineage query ecosystem
airflow-lineage draw --ecosystem --open
Point it at your own repository the same way:
airflow-lineage extract --dag-dir path/to/dags --var-dir path/to/variables
--var-dir is optional. Without it, values that come from Airflow Variables stay
unresolved (and are reported as such) — everything else still works.
Exporting your Airflow Variables
The extractor reads a folder of JSON files shaped {"<variable_name>": <value>},
which is what airflow variables export produces per key:
airflow variables export variables.json
Split it into one file per key, or drop the single file in the folder — both are supported.
Commands
airflow-lineage extract Parse DAG files -> dag_lineage.json
airflow-lineage query Ask dependency questions about the graph
airflow-lineage draw Render an interactive HTML graph
airflow-lineage audit Fail the build on graph-integrity errors
airflow-lineage export Flatten to CSV / JSON / XLSX
airflow-lineage ui Serve the local web UI
Useful queries:
airflow-lineage query impact <dag> # blast radius, transitive
airflow-lineage query depends <dag> # everything it waits for
airflow-lineage query path <from> <to> # shortest route between two DAGs
airflow-lineage query hubs --top 10 # most-connected DAGs
airflow-lineage query cycles # circular dependencies
airflow-lineage query operator DatabricksSubmitRunOperator
airflow-lineage query find-task "load_.*"
airflow-lineage query export-mermaid <dag> --depth 2
Add --json to any query for machine-readable output. Full list: docs/cli.md.
Web UI
pip install "airflow-lineage-toolkit[ui]"
airflow-lineage ui --dag-dir examples/dags --var-dir examples/variables
--dag-dir and --var-dir default to ./dags and ./variables relative to
wherever you run the command — if that doesn't match your layout, the Run
extraction button in the UI will fail with DAG directory not found. Pass
the real paths explicitly, as above. Full options: docs/cli.md.
Use it in CI
Catch a broken graph in review instead of at 3am:
- run: pip install airflow-lineage-toolkit
- run: airflow-lineage extract --dag-dir dags --var-dir variables --quiet
- run: airflow-lineage audit # non-zero exit on phantom/disconnected edges
Use it as a library
from airflow_lineage import build_lineage, LineageGraph
report = build_lineage("dags", "variables")
graph = LineageGraph(report)
print(graph.transitive_down("orders_ingestion_daily")) # blast radius
print(graph.shortest_paths("raw_load", "exec_dashboard"))
The output document schema is described in docs/lineage-schema.md.
How it works
A per-file ast.NodeVisitor runs several passes: custom sensor classes, operator
factories, JSON payload builders, a fix-point over module-level assignments (so
forward references resolve), store-time Variable producers/consumers, and finally
the DAG/task/edge walk. Loops and comprehensions are executed statically against
resolved config so dynamically generated tasks land in the graph as concrete nodes
with their edges. See docs/how-it-works.md for the
design notes and the failure modes it was hardened against.
Limitations
- Task ids that depend on runtime state (XComs,
{{ ds }}, API calls) cannot be resolved — they are reported as unresolved rather than invented. - TaskFlow (
@task) decorators are not yet mapped to task ids. - Operator action detail currently covers Databricks and dbt Cloud; other operators are captured as nodes with their operator name only.
Contributions in these areas are very welcome — see CONTRIBUTING.md.
License
MIT
Metadata
Release files for airflow-lineage-toolkit 1.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| airflow_lineage_toolkit-1.0.0.tar.gz | 68.9 kB | Without provenance | Details for airflow_lineage_toolkit-1.0.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| airflow_lineage_toolkit-1.0.0-py3-none-any.whl | Python 3 | none | any | Details for airflow_lineage_toolkit-1.0.0-py3-none-any.whl |
Total release size: 132.3 kB
Release files / airflow_lineage_toolkit-1.0.0.tar.gz
| Download URL | airflow_lineage_toolkit-1.0.0.tar.gz |
|---|---|
| Size | 68.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
403f824c8722caa2f3d1c256b36d8753c4d2b83cb62256f922e53ffbe7f2861c
|
|
BLAKE2b-256 checksum How to use checksums |
a26525704d1994bf89f2235a63f06a749ed7888372360c5e3b928ddfbee6cad9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.10
|
Release files / airflow_lineage_toolkit-1.0.0-py3-none-any.whl
| Download URL | airflow_lineage_toolkit-1.0.0-py3-none-any.whl |
|---|---|
| Size | 63.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1a5dfa6d737fb2a6c9cbb17a1cda0c7f386b0e56c708fe149f8b8c0124a66708
|
|
BLAKE2b-256 checksum How to use checksums |
1210542fc1b7d7f4caffdac1fb227ce3899562cb81a3e362ce70a8d1a170c6e0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.10
|
Security for airflow-lineage-toolkit 1.0.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.