Skip to main content

airflow-lineage-toolkit

Answer "what breaks if I change this DAG?" without running Airflow.

airflow-lineage-toolkit reads your Airflow DAG source files with Python's ast module and builds a complete dependency graph: every task, every >>, every cross-DAG sensor and trigger, plus what each task actually runs (Databricks main class / JAR / notebook / wheel, dbt Cloud job id).

No Airflow instance. No metadata database. No importing your DAG code. Just the standard library pointed at a folder.

pip install airflow-lineage-toolkit

airflow-lineage extract --dag-dir dags --var-dir variables
airflow-lineage query impact orders_ingestion_daily
airflow-lineage draw orders_ingestion_daily --open

Why

Most lineage tooling needs a live scheduler, a plugin, or a running task to emit events. That is useless when the question is asked before the change lands:

  • Which DAGs break if I retire this one?
  • What is the full upstream chain behind this table?
  • Which tasks still point at that legacy JAR?
  • Is my hand-drawn architecture diagram still true?

Static analysis answers all of those from source, in seconds, in CI, on a laptop, on a branch that has never been deployed.

What it does

Intra-DAG lineage >>, <<, set_upstream/downstream, chain(), list fan-out/fan-in
Cross-DAG lineage ExternalTaskSensor, TriggerDagRunOperator, custom sensor classes, and producer/consumer edges inferred from Variable.set/Variable.get execution-time markers
Dynamic tasks tasks built in for loops, comprehensions, factory functions, range/enumerate/zip, f-string and .format() task ids
Config resolution reads a directory of exported Airflow Variable JSON files, so Variable.get("cfg", deserialize_json=True)["tables"] resolves to real values
Conditional branches evaluates if/elif/else statically and captures only the branch that is actually active
Task actions Databricks main_class, JAR, notebook path, python file, wheel package/entry point, parameters; dbt Cloud job_id
Integrity audit fails loudly on phantom edges, disconnected dynamic tasks, self-loops, unresolved template ids

Truthfulness over completeness

A value that cannot be resolved statically is left null and surfaced under unresolved_references — never guessed. Shape guards reject anything that looks like leaked source text masquerading as a resolved value (main_class must be a valid dotted class name; paths must not contain brackets or quotes). A wrong "resolved" value is worse than an honest gap, especially when the output feeds a migration inventory.

Install

pip install airflow-lineage-toolkit                 # core: standard library only
pip install "airflow-lineage-toolkit[ui]"           # + local web UI (FastAPI)
pip install "airflow-lineage-toolkit[excel]"        # + .xlsx export (openpyxl)

Python 3.9+. The core extractor, query CLI, HTML visualizer, auditor and CSV/JSON export have zero runtime dependencies — they run in any CI image.

Quickstart

The repository ships a small synthetic Airflow project under examples/ that exercises every hard pattern, so you can see real output immediately:

airflow-lineage extract --dag-dir examples/dags --var-dir examples/variables
airflow-lineage audit
airflow-lineage query ecosystem
airflow-lineage draw --ecosystem --open

Point it at your own repository the same way:

airflow-lineage extract --dag-dir path/to/dags --var-dir path/to/variables

--var-dir is optional. Without it, values that come from Airflow Variables stay unresolved (and are reported as such) — everything else still works.

Exporting your Airflow Variables

The extractor reads a folder of JSON files shaped {"<variable_name>": <value>}, which is what airflow variables export produces per key:

airflow variables export variables.json

Split it into one file per key, or drop the single file in the folder — both are supported.

Commands

airflow-lineage extract   Parse DAG files -> dag_lineage.json
airflow-lineage query     Ask dependency questions about the graph
airflow-lineage draw      Render an interactive HTML graph
airflow-lineage audit     Fail the build on graph-integrity errors
airflow-lineage export    Flatten to CSV / JSON / XLSX
airflow-lineage ui        Serve the local web UI

Useful queries:

airflow-lineage query impact <dag>            # blast radius, transitive
airflow-lineage query depends <dag>           # everything it waits for
airflow-lineage query path <from> <to>        # shortest route between two DAGs
airflow-lineage query hubs --top 10           # most-connected DAGs
airflow-lineage query cycles                  # circular dependencies
airflow-lineage query operator DatabricksSubmitRunOperator
airflow-lineage query find-task "load_.*"
airflow-lineage query export-mermaid <dag> --depth 2

Add --json to any query for machine-readable output. Full list: docs/cli.md.

Web UI

pip install "airflow-lineage-toolkit[ui]"
airflow-lineage ui --dag-dir examples/dags --var-dir examples/variables

--dag-dir and --var-dir default to ./dags and ./variables relative to wherever you run the command — if that doesn't match your layout, the Run extraction button in the UI will fail with DAG directory not found. Pass the real paths explicitly, as above. Full options: docs/cli.md.

Use it in CI

Catch a broken graph in review instead of at 3am:

- run: pip install airflow-lineage-toolkit
- run: airflow-lineage extract --dag-dir dags --var-dir variables --quiet
- run: airflow-lineage audit          # non-zero exit on phantom/disconnected edges

Use it as a library

from airflow_lineage import build_lineage, LineageGraph

report = build_lineage("dags", "variables")
graph = LineageGraph(report)

print(graph.transitive_down("orders_ingestion_daily"))   # blast radius
print(graph.shortest_paths("raw_load", "exec_dashboard"))

The output document schema is described in docs/lineage-schema.md.

How it works

A per-file ast.NodeVisitor runs several passes: custom sensor classes, operator factories, JSON payload builders, a fix-point over module-level assignments (so forward references resolve), store-time Variable producers/consumers, and finally the DAG/task/edge walk. Loops and comprehensions are executed statically against resolved config so dynamically generated tasks land in the graph as concrete nodes with their edges. See docs/how-it-works.md for the design notes and the failure modes it was hardened against.

Limitations

  • Task ids that depend on runtime state (XComs, {{ ds }}, API calls) cannot be resolved — they are reported as unresolved rather than invented.
  • TaskFlow (@task) decorators are not yet mapped to task ids.
  • Operator action detail currently covers Databricks and dbt Cloud; other operators are captured as nodes with their operator name only.

Contributions in these areas are very welcome — see CONTRIBUTING.md.

License

MIT

Metadata

Release files for airflow-lineage-toolkit 1.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for airflow-lineage-toolkit 1.0.0
File Size Uploaded Provenance Actions
airflow_lineage_toolkit-1.0.0.tar.gz 68.9 kB Without provenance Details for airflow_lineage_toolkit-1.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for airflow-lineage-toolkit 1.0.0
File Interpreter ABI Platform Actions
airflow_lineage_toolkit-1.0.0-py3-none-any.whl Python 3 none any Details for airflow_lineage_toolkit-1.0.0-py3-none-any.whl

Total release size: 132.3 kB

Release files / airflow_lineage_toolkit-1.0.0.tar.gz

Download URL airflow_lineage_toolkit-1.0.0.tar.gz
Size 68.9 kB
Tags Source
SHA-256 checksum
How to use checksums
403f824c8722caa2f3d1c256b36d8753c4d2b83cb62256f922e53ffbe7f2861c
BLAKE2b-256 checksum
How to use checksums
a26525704d1994bf89f2235a63f06a749ed7888372360c5e3b928ddfbee6cad9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.10

Release files / airflow_lineage_toolkit-1.0.0-py3-none-any.whl

Download URL airflow_lineage_toolkit-1.0.0-py3-none-any.whl
Size 63.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
1a5dfa6d737fb2a6c9cbb17a1cda0c7f386b0e56c708fe149f8b8c0124a66708
BLAKE2b-256 checksum
How to use checksums
1210542fc1b7d7f4caffdac1fb227ce3899562cb81a3e362ce70a8d1a170c6e0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.10

Security for airflow-lineage-toolkit 1.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.0.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page