DBT Column Lineage

📖 Documentation | 🚀 Live Demo | 🐛 Report Bug | 💡 Request Feature

Overview

Understand the impact of column changes before you make them. DBT Column Lineage provides impact analysis & column lineage to help you answer critical questions: What happens if I change this column? Which models, transformations, and exposures will be affected? Does it use complex business logic, or is this column only projected?

The core idea behind this project: be more comfortable when making changes on a large project by knowing the impact.

Key Features:

• Impact Analysis: Understand downstream effects of column changes, including affected models, transformations, and exposures
• Interactive Explorer: A local web server with an intuitive UI to explore model and column lineage visually, including associated transformations

Installation

pip install dbt-col-lineage

Usage

First, ensure your dbt project is compiled and you have generated the catalog:

dbt compile
dbt docs generate

Interactive Explorer

Start the interactive web interface to explore lineage and perform impact analysis:

dbt-col-lineage --explore \
    --manifest path/to/manifest.json \
    --catalog path/to/catalog.json \
    --port 8080  # Optional port selection

This starts a server (defaulting to port 8000). Open your web browser to the specified address (e.g., http://127.0.0.1:8080).

Workflow:

1. Select a model and column from the sidebar to visualize its lineage
2. Click "Analyze Impact" to see which models, transformations, and exposures will be affected if you change that column
3. Review the impact analysis panel showing:
   • Columns that require review (with transformations that may break)
   • Pass-through columns (direct references)
   • Affected models in the dependency chain
   • Affected exposures

Machine-readable output (AI / automation)

Emit column lineage and downstream impact analysis as a single JSON document — ideal for feeding AI agents, CI checks, or other tooling:

dbt-col-lineage --select stg_accounts.account_id --format json \
    --manifest path/to/manifest.json --catalog path/to/catalog.json

The document contains the column's upstream and downstream lineage (each split into models, sources, direct_refs, exposures) and, when downstream lineage is requested, an impact block summarising affected models, columns and exposures.

Note: the tool works even when your manifest.json doesn't embed compiled_code (e.g. produced by dbt parse), as long as target/compiled/** exists — it falls back to the compiled SQL on disk.

Diff-driven impact (impact)

The unit of work in a PR is a set of changed columns, not a single one. The impact command derives that changeset and reports one consolidated blast radius.

Two-manifest diff (the reliable, dbt-native signal — compares the base branch's artifacts against the current ones for added / removed / retyped columns and changed compiled SQL):

dbt-col-lineage impact \
    --manifest target/manifest.json --catalog target/catalog.json \
    --base-manifest base/manifest.json --base-catalog base/catalog.json

Git-diff fallback (when only one manifest is available — diffs changed .sql model files against a git ref and treats touched models as logic changes):

dbt-col-lineage impact --git-base main

Output defaults to a human-readable Markdown summary (exposures first, then a blast-radius table). Use --format json for a machine-readable report — a superset of the single-column impact block plus a top-level changeset and a per-change by_change breakdown.

CI check + PR comment (impact --ci)

--ci turns the report into a pull-request check: it posts a sticky Markdown comment (found-or-updated by a hidden marker, so re-runs edit one comment instead of spamming the thread) and applies a severity gate as the process exit code.

dbt-col-lineage impact --ci \
    --manifest target/manifest.json --catalog target/catalog.json \
    --base-manifest base/manifest.json --base-catalog base/catalog.json \
    --fail-on exposures

• --fail-on: gate policy — none (default: warn only, never block), exposures (fail when a business-facing exposure is affected), critical (fail when a downstream column recomputes derived logic), or any.
• The PR context (token, repo, PR number) is resolved from the standard GitHub Actions environment (GITHUB_TOKEN, GITHUB_REPOSITORY, GITHUB_EVENT_PATH); override with --github-token / --repo / --pr-number. When no context is available the comment is skipped and only the gate runs.

GitHub Action. A ready-to-use composite action ships in this repo. Pin the floating major tag to get updates within a major version (the usual convention, like actions/checkout@v4); the tool is currently 0.x, so that tag is @v0:

- uses: Fszta/dbt-column-lineage@v0
  with:
    manifest: artifacts/head/manifest.json
    catalog: artifacts/head/catalog.json
    base-manifest: artifacts/base/manifest.json
    base-catalog: artifacts/base/catalog.json
    fail-on: none # start non-blocking; flip to exposures|critical once trusted

For reproducible builds, pin an exact release instead — @v0.8.0 (the same tag as the pip package). The action installs the source bundled at whichever ref you pin, so the CLI always matches it.

The workflow needs permissions: pull-requests: write to post the comment. A complete example workflow (building base- and head-branch artifacts) lives at docs/examples/impact-pr-check.yml.

Options

• --explore: Starts the interactive web server for exploring lineage and impact analysis
• --catalog: Path to the dbt catalog file (default: target/catalog.json)
• --manifest: Path to the dbt manifest file (default: target/manifest.json)
• --port, -p: Port for the interactive web server (default: 8000)
• --adapter: Override the SQL dialect used by the parser (sqlglot dialect name, e.g., tsql, snowflake, bigquery). When provided, this overrides the adapter detected from the dbt manifest.
• --format, -f: Output format for --select: text (default), dot, or json.

For the impact subcommand:

• --base-manifest / --base-catalog: base-branch artifacts for the two-manifest diff.
• --git-base: git ref to diff changed .sql files against (git-diff fallback).
• --scope-git: with a two-manifest diff, restrict the report to models changed in git diff <ref>...HEAD — drops changes on models the branch didn't touch (useful when the base artifacts may be staler than the base ref).
• --ci: post a sticky PR comment and apply the --fail-on severity gate as an exit code.
• --fail-on: gate policy — none (default), exposures, critical, or any.
• --github-token / --repo / --pr-number: override the GitHub context (defaults from the Actions env).

Limitations

• Doesn't support python models
• Some functions/syntax cannot be parsed properly, leading to models being skipped

Compatibility

The tool has been tested with the following dbt adapters:

• Snowflake
• SQLite
• DuckDB
• MS SQLServer / TSQL

License

This project is licensed under the MIT License - see the LICENSE file for details.