lineagemap 0.1.0

Column-level data lineage for dbt projects. One command, no catalog required.

LineageMap

Column-level data lineage for dbt projects. One command, no catalog required.

$ lineagemap trace --column revenue

revenue (fct_orders)
└── revenue (stg_orders)
    └── amount (raw.orders)

Install

pip install lineagemap

For the web UI:

pip install "lineagemap[server]"

Usage

Run dbt compile first, then:

# Interactive web UI at http://localhost:3000
lineagemap serve

# Trace a column in the terminal
lineagemap trace --column revenue

# Scope to a specific model
lineagemap trace --column revenue --model fct_orders

# Export full lineage graph as JSON
lineagemap trace --column revenue --json

# Custom manifest path or SQL dialect
lineagemap trace --column revenue --manifest path/to/manifest.json --dialect snowflake

Web UI

Hover any node to see its full upstream (blue) and downstream (red) lineage instantly. Use the search bar to filter by column or model name.

Self-hosting

LineageMap can be self-hosted so your whole team can access it in a browser, without anyone needing to install Python or run dbt compile locally.

Prerequisites

• Docker installed
• A manifest.json from your dbt project (generated by dbt compile)

Option 1: Docker (quickest)

docker run -p 3000:3000 \
  -v $(pwd)/manifest.json:/manifest.json:ro \
  ghcr.io/christianquebral/lineagemap:latest

Then open http://localhost:3000.

To use a different port:

docker run -p 8080:3000 \
  -v $(pwd)/manifest.json:/manifest.json:ro \
  ghcr.io/christianquebral/lineagemap:latest

Option 2: Docker Compose (recommended for persistent deploys)

1. Copy docker-compose.yml from this repo into your project directory
2. Run dbt compile to generate manifest.json
3. Start the server:

docker compose up -d

To point at a manifest in a different location:

MANIFEST_PATH=/path/to/your/manifest.json docker compose up -d

To use a different port:

PORT=8080 docker compose up -d

Stop the server:

docker compose down

Option 3: pip install

Install the package, then serve directly from your dbt project directory:

pip install "lineagemap[server]"
dbt compile
lineagemap serve --host 0.0.0.0 --manifest target/manifest.json

Share http://<your-server-ip>:3000 with your team.

By default (--host 127.0.0.1) the server is localhost-only. Pass --host 0.0.0.0 to make it network-accessible.

To keep it running after you close the terminal:

# Using nohup
nohup lineagemap serve --host 0.0.0.0 --manifest target/manifest.json &

# Or with screen
screen -S lineagemap
lineagemap serve --host 0.0.0.0 --manifest target/manifest.json
# Detach with Ctrl+A, D

For a more robust setup (auto-restart on reboot), use Docker Compose (Option 2) or configure a systemd service.

Keeping lineage up to date automatically

LineageMap exposes a POST /api/reload endpoint that re-parses the manifest and updates every connected browser — no restart needed.

1. Set a reload token on your server

This protects the endpoint so only your CI pipeline can trigger a reload.

In your .env or shell:

export LINEAGEMAP_RELOAD_TOKEN=your-secret-token

With Docker Compose:

LINEAGEMAP_RELOAD_TOKEN=your-secret-token docker compose up -d

2. Trigger a reload from CI

After dbt compile runs in your dbt project's CI pipeline, add a step to notify LineageMap:

curl -X POST https://lineagemap.your-company.com/api/reload \
  -H "Authorization: Bearer your-secret-token"

Or use the ready-made GitHub Action — copy .github/workflows/lineage-sync.yml from this repo into your dbt project and add two repository secrets:

Secret	Value
LINEAGEMAP_URL	http://lineagemap.your-company.com:3000
LINEAGEMAP_RELOAD_TOKEN	your secret token

The workflow fires automatically on pushes to main that touch models/ or dbt_project.yml.

Protecting access with basic auth

To require a username and password before anyone can view the dashboard, use the included nginx overlay.

1. Generate a password file

# macOS
brew install httpd && htpasswd -c .htpasswd <username>

# Linux
apt install apache2-utils && htpasswd -c .htpasswd <username>

2. Start with auth enabled

docker compose -f docker-compose.yml -f docker-compose.auth.yml up -d

Your team accesses http://your-server:3000 and is prompted for credentials. The /api/reload endpoint bypasses basic auth so CI can still trigger reloads using only the Bearer token.

How it works

1. Reads manifest.json from dbt compile (no live warehouse connection needed)
2. Parses each model's compiled SQL with sqlglot to extract column-level dependencies
3. Builds a column lineage graph you can explore in the terminal or web UI

Why LineageMap

Honest landscape note: dbt Core v2.0 (Fusion) now ships column-level lineage in the OSS core, and dbt-column-lineage covers similar ground with the same sqlglot approach. The CLI problem is largely solved.

What remains unoccupied: a polished web UI and a hosted/shareable tier (upload a manifest, send a stakeholder a link, no local setup required). That is what LineageMap is building toward.

	LineageMap	dbt Fusion	dbt-column-lineage	dbt Cloud	DataHub
Column-level lineage	✅	✅	✅	✅	✅
One-command setup	✅	✅ (built-in)	✅	N/A	❌
Interactive web UI	✅	❌	❌	✅	✅
Shareable URL	Coming (v0.3)	❌	❌	✅	❌
No warehouse connection	✅	✅	✅	✅	❌
Self-hostable	✅	✅	✅	❌	✅ (complex)
Cost	Free	Free	Free	$$$	Free (DIY)

Roadmap

• Phase 1: CLI + SQL parser (lineagemap trace)
• Phase 2: Local web UI with interactive DAG visualization
• Phase 3: Hosted tier: upload manifest.json → get a shareable URL
• Phase 4: GitHub Action: post updated lineage URL on every dbt PR

Contributing

See CONTRIBUTING.md. Good first issues are tagged in the tracker.

License

Business Source License 1.1. Converts to Apache 2.0 on 2030-06-26. Free for personal, internal, and non-competing production use.