datagraph

The open-source data engine: lineage · relationships · data profiling · dimensional modelling · knowledge graph for AI assistants
built deterministically from your database, dbt project, SQL and code — no LLM in the loop.

pip install "datagraph-core[sql]"
datagraph analyze --warehouse "snowflake://user:pw@account/db" --schemas analytics -o out/
#  -> lineage.html · relationships.json · MODEL.md + er-diagram.mmd
#     profiles · wiki/ (for AI assistants) · datagraph.json

Brownfield data platform, hundreds of tables, no documentation? One command turns the schema (and dbt / SQL / code if you have them) into a graph you can query: where does this come from, what does it feed, how are the tables related, what does the data look like, what is the dimensional model, what breaks if I change this — and hands the same graph to your AI assistant over MCP.

How it works

Real output: datagraph lineage customers --html on dbt's public jaffle_shop project — upstream through staging models to seed files, downstream to the table and its columns.

Why datagraph

Principles: the graph is built deterministically from artifacts (never by an LLM) · every edge carries provenance (extracted / inferred / llm) and --no-inferred strips heuristics · nothing to deploy — pip, one JSON file, runs in CI · secrets never stored, personal data masked · LLMs are optional and only explain.

New here? Run the guided tour - it builds a demo warehouse and walks through every feature with real output:

python examples/example_datagraph.py

Contents

1. Install
2. The standard flow: connection in → lineage, profiling, model out
3. What goes into the graph
4. Commands
5. Dimensional modelling
6. Data profiling
7. Knowledge base & MCP for AI assistants
8. Impact analysis & the impactgraph companion
9. Python API
10. Security
11. How it compares
12. Node ids, provenance, propagation
13. Development & roadmap

Install

pip install datagraph-core              # graph, extractors, lineage, profiling, modelling
pip install "datagraph-core[sql]"       # + sqlglot: SQL and column lineage (recommended)
pip install "datagraph-core[mcp]"       # + MCP server for coding assistants
pip install "datagraph-core[ai]"        # + Anthropic Claude (explanations, fallback)
pip install "datagraph-core[bedrock]"   # + Amazon Bedrock (Nova, Claude, Llama)
pip install "datagraph-core[all]"       # everything (adds PyYAML, mcp, boto3)

The PyPI distribution is datagraph-core (the bare name datagraph is not allowed on PyPI); the import name, CLI and MCP server are all datagraph: pip install datagraph-core → import datagraph / datagraph analyze ….

Database drivers: SQLite and DuckDB files work out of the box; for Snowflake / Postgres / BigQuery / Redshift / MySQL / SQL Server install SQLAlchemy plus the driver and pass a SQLAlchemy URL (or pass an open DB-API connection from Python). From source: pip install "datagraph-core[sql] @ git+https://github.com/sumit-gupta03/datagraph".

The standard flow: connection in → lineage, profiling, model out

datagraph analyze --warehouse "snowflake://user:pw@account/db" -o out/
datagraph analyze --warehouse warehouse.db -o out/     # SQLite / DuckDB file

One command runs the standard sequence (use a read-only database role; the password is never stored or logged):

Then ask questions against the saved graph:

datagraph lineage fact_sales --graph out/datagraph.json     # upstream + downstream
datagraph relationships --graph out/datagraph.json --search customer
# compact knowledge pack for an assistant
datagraph context dim_customer --graph out/datagraph.json
datagraph model --graph out/datagraph.json --from-table wide_orders
# MCP server for your coding assistant
datagraph mcp --graph out/datagraph.json

Options: --schemas a,b · --database NAME · --dialect snowflake|postgres|bigquery|… (for view SQL) · --no-profile (metadata only) · --sample N · --no-top-values · --no-inferred (declared foreign keys only) · --json.

What goes into the graph

datagraph build accepts any combination; fragments merge by shared node ids and table aliases (analytics.orders vs prod.analytics.orders) are linked automatically.

Commands

Every command takes --graph PATH (default datagraph.json), most take --json and --no-inferred.

Dimensional modelling

datagraph model                      # classification, star schema, issues, ER diagram
datagraph model --markdown MODEL.md --mermaid er.mmd --json
# propose fact + dimensions from one flat / wide table
datagraph model --from-table wide_orders
datagraph model --no-inferred                     # declared foreign keys only

Real output of datagraph model --mermaid on a small warehouse (fact + three dimensions, SCD type 2 detected on dim_product):

…and the matching report:

## Bus matrix (facts x dimensions)
| fact       | dim_customer | dim_product | dim_date |
|------------|--------------|-------------|----------|
| fact_sales | X            | X           | X        |

### table:fact_sales - fact (confidence 0.95)
- grain: date_key, customer_id, product_id        - measures: amount, quantity
- Kimball: process `sales` -> grain: date_key x customer_id x product_id
           -> 3 dimensions -> 2 measures

## Dimensions
- dim_customer - key customer_id, 4 attribute(s), used by fact_sales; SCD type 1
- dim_product  - key product_id,  5 attribute(s), used by fact_sales; SCD type 2
- dim_date     - key date_key,    3 attribute(s), used by fact_sales; SCD type static

Standard Kimball approach, computed deterministically and explained (every classification lists its reasons):

• Column roles — pk / fk / date / measure / flag / attribute from names, declared types (warehouse or dbt catalog) and profiles.
• Table roles — fact / dimension / bridge / lookup / derived (views without key links) with a confidence and reasons: foreign keys out/in, measures, dates, attributes, naming conventions, row counts.
• Key links — declared foreign keys (extracted) plus name inference such as orders.customer_id → customers (inferred, flagged to verify).
• Star schema — per fact: business process → grain → dimensions → facts (the four-step design), measures with additivity; per dimension: key, attributes, used-by, SCD type (2 when valid_from/valid_to/is_current exist, 1 when updated_at, else undecided with a recommendation); bus matrix (facts × dimensions) and conformed dimensions; snowflake chains.
• Issues — fact without a time grain, key with no dimension, fact-to-fact links, measures sitting in a dimension, unused dimensions, natural/text keys (surrogate key advice), missing dim_date, high-null keys (late-arriving dimensions).
• Propose from a wide table — groups low-cardinality attributes by prefix into dimensions (customer_name, customer_country → dim_customer), numeric columns into measures, dates into dim_date; near-unique text stays as degenerate dimensions.

MODEL.md is part of the wiki, model is an MCP tool, and the role shows up in context packs.

Data profiling

datagraph profile --warehouse prod.db [--tables customers,orders] [--sample 100000]

Per table: row count, freshness (max of date-like columns); per column: null %, distinct, min/max, top values (sampled). Results are stored on the graph nodes and surface in relationships, context, lineage HTML tooltips and the wiki. Columns whose names look sensitive (email, phone, name, address, card, token, …) keep counts only — no sample values. Profiles also make the risk score data-aware (empty tables count half, >1M-row tables 1.5×) and feed the optional LLM lineage fallback.

Governance: glossary, domains, deprecation, test results, PII

Catalog concepts live in one file you commit - no database, no UI behind them:

# datagraph.yml   ->   datagraph build ... --metadata datagraph.yml
glossary:
  - term: Customer PII
    definition: Personal data about an identified customer.
    applies_to: ["column:dim_customer.email", "column:dim_customer.name"]
domains:
  - name: Finance
    owner: finance
    assets: ["dbt:fact_*", "table:prod.analytics.*"]
deprecations:
  - asset: dbt:legacy_customer
    replacement: dbt:dim_customer

A dbt project needs no file at all: meta.domain, group, meta.terms and meta.deprecated (or a deprecated tag) are read straight from the manifest.

# names, ids, descriptions, columns, owners, tags, terms
datagraph search customer
datagraph search --domain Finance --type dbt_model
datagraph glossary                   # terms, definitions, and the assets carrying them
datagraph pii                        # personal data, and what is exposed to it

Test outcomes and source freshness come from dbt's own artifacts, auto-detected next to the manifest or passed explicitly:

datagraph build --dbt-manifest target/manifest.json \
                --dbt-run-results target/run_results.json \
                --dbt-sources target/sources.json

Impact analysis then warns about what it finds:

! 'dim_customer' has 1 failing dbt test(s): not_null_dim_customer_customer_id
! source 'raw.customers' freshness is warn (last loaded 2026-08-20T00:00:00Z)
! 'customer' is deprecated - use dbt:dim_customer instead

The same facts reach GRAPH_REPORT.md (deprecated assets still in use, failing tests, stale sources, domains, glossary, sensitive-data map), the wiki index (grouped by domain) and MCP (search and sensitive_data tools).

Browse it locally: datagraph serve

datagraph serve --graph datagraph.json --open      # http://127.0.0.1:8765

A single-user, read-only browser for the graph, served by Python's own HTTP server - no database, no accounts, no write path, no extra dependency. Search as you type, filter by type and domain, open an asset to see its full context pack (columns, profile, usage, lineage, risk, SQL), and click through to the interactive lineage or whole-graph views. It binds to loopback only unless you pass --host (and warns if you do), and re-reads the graph file when it changes, so a rebuild in another terminal just needs a refresh.

Routes: / · /api/search · /api/node/<id> · /api/report · /api/model · /lineage/<id> · /graph.

Usage: what is actually queried, and what can be deleted

datagraph usage --warehouse "snowflake://..." --days 30
datagraph usage --warehouse "postgresql+psycopg2://..." --unused-only
# collect it as part of the standard flow
datagraph analyze --warehouse ... --usage

Reads the engine's own query log - Snowflake ACCESS_HISTORY, BigQuery INFORMATION_SCHEMA.JOBS, PostgreSQL pg_stat_user_tables, MySQL performance_schema - and attaches counts to the graph. Engines without a query log are reported and skipped; a missing grant is reported, never fatal.

The payoff for a brownfield warehouse is the intersection of two signals:

never queried (3):
  table:legacy_scratch, 0 rows - safe to drop (nothing downstream)
  table:old_orders_backup, 1.2M rows - safe to drop (nothing downstream)
  table:stg_legacy - 2 downstream

"Nobody queries it" and "nothing depends on it" is the only combination that is actually safe, and datagraph is holding both halves. Usage also appears in context, the wiki's GRAPH_REPORT.md ("Never queried") and the usage MCP tool.

Knowledge base & MCP for AI assistants

datagraph context dim_customer       # owner, columns (+type, pk, profile, sources),
                                     # upstream, downstream, relationships, tests,
                                     # risk if changed, test plan, and the SQL
datagraph wiki -o kb/                # index.md, nodes/*.md, GRAPH_REPORT.md, llms.txt

GRAPH_REPORT.md lists hotspots, high-impact dbt models without tests, ownerless nodes, roots and leaves. Everything is generated from the graph, so an assistant explains rather than guesses.

MCP (Claude Code, Claude Desktop, Cursor — any MCP client), after pip install "datagraph-core[mcp]" and one analyze/build:

{
  "mcpServers": {
    "datagraph": {
      "command": "python",
      "args": ["-m", "datagraph.cli", "mcp", "--graph", "/path/to/out/datagraph.json"]
    }
  }
}

(examples/mcp/claude-mcp.json; for Claude Code put it in .mcp.json or run claude mcp add datagraph -- python -m datagraph.cli mcp --graph /path/to/datagraph.json.) Tools: impact, diff, find_nodes, paths, hotspots, lineage, relationships, context, model. The server is stdio-only, read-only over the graph file you pass, and never receives connection strings.

Claude Code skill: copy skills/datagraph/ to .claude/skills/datagraph/ (or ~/.claude/skills/) and ask "where does fact_booking come from?", "how are these tables related?", "what is the dimensional model?", "what breaks if I change dim_customer?".

Impact analysis & the impactgraph companion

# a model / table / column / function / task
datagraph impact dbt:customer
datagraph diff --repo . --graph datagraph.json   # what my uncommitted change can break
datagraph paths dbt:customer exposure:revenue_report
datagraph hotspots

⚠ Change Impact                     Changed: customer      Risk: HIGH (score 24.5)

⬢ customer (dbt_model)
├── ⬢ dim_customer (dbt_model) via depends_on
│   └── ⬢ fact_booking (dbt_model) via depends_on
│       ├── 📊 revenue_report (dashboard) via exposes
│       └── 📊 customer_dashboard (dashboard) via exposes
└── ▤ prod.analytics.customer (view) via writes_to

Affected: 3 dbt model(s) · 2 dashboard(s) · 2 table(s)
Notify (owners of affected artifacts):  finance: revenue_report · growth: customer_dashboard
Recommended tests:
  ✓ dbt build --select customer+ dim_customer+ fact_booking+
  ✓ Run a schema/contract check on prod.analytics.fact_booking
  ✓ Manually validate 'revenue_report' after deploy (numbers & filters)

datagraph html models/customer.sql — one SQL file → models → tables → dashboards and the Python API, with risk, owners and the test plan.

The pull-request product — impactgraph check / pr, a GitHub Action that comments the blast radius on every PR, --fail-on gating — lives in impactgraph, a thin layer over this engine that re-exports its whole API. datagraph = everything data-related; impactgraph = "what breaks if I merge this?".

Python API

from datagraph import (ImpactGraph, WarehouseExtractor, DbtExtractor,
                       SqlExtractor, PythonExtractor, AirflowExtractor,
                       LambdaExtractor, JsExtractor, OpenLineageExtractor,
                       LineageFileExtractor, DataHubExtractor, analyze_impact,
                       profile_warehouse, star_schema, propose_from_table, classify_tables,
                       context, build_wiki, ExtractorPlugin, register)

# 1. build (any combination; a DSN, a file path or an open DB-API connection)
graph = ImpactGraph()
graph.merge(WarehouseExtractor("snowflake://...", schemas=["analytics"]).extract())
graph.merge(DbtExtractor("target/manifest.json").extract())
graph.merge(PythonExtractor("./src").extract())
graph.link_table_aliases()

# 2. lineage & relationships
# {'upstream': {...}, 'downstream': {...}}
graph.lineage("table:analytics.dim_customer")
from datagraph.analysis.relationships import relationships
# foreign keys + lineage between tables
relationships(graph)["table_relationships"]

# 3. profiling, dimensional model, knowledge base
# stores node.meta["profile"] (sensitive columns masked)
profile_warehouse("snowflake://...", graph)
# facts, dimensions, bus_matrix, scd, issues
model = star_schema(graph)
from datagraph.analysis.modeling import to_markdown, to_mermaid
print(to_markdown(model)); print(to_mermaid(model))
propose_from_table(graph, "wide_orders")                  # star from a flat table
print(context(graph, "dim_customer"))                     # compact text pack
build_wiki(graph, "kb/")

# 4. impact
analysis = analyze_impact(graph, ["dbt:customer"])
analysis.risk, analysis.owners, analysis.recommended_tests, analysis.trees

# 5. your own extractor -> also becomes `datagraph build --mytool X`
register(ExtractorPlugin(name="mytool", factory=MyToolExtractor,
                         options={"token": "API token"}))
# or via an entry point:  [project.entry-points."datagraph.extractors"]
#                         mytool = "my_pkg:MyToolExtractor"

# 6. optional AI (pip install datagraph-core[ai])
from datagraph.ai import explain_impact, suggest_lineage, apply_suggestions
# explains; never changes the graph
print(explain_impact(analysis))
# tagged provenance=llm, excludable
apply_suggestions(graph, suggest_lineage(graph), min_confidence=0.7)

Optional AI layer and LLM providers

The AI layer is optional and never builds the graph: datagraph explain narrates an impact analysis, datagraph enrich / build --llm-fallback asks for relationship suggestions (schema-validated, must reference existing nodes, tagged llm, confidence-gated). Three interchangeable providers; pick with --provider or DATAGRAPH_LLM_PROVIDER, model with --model or DATAGRAPH_LLM_MODEL; credentials always come from the environment / cloud SDK, never from the graph:

from datagraph.ai import explain_impact, suggest_lineage, BedrockProvider
print(explain_impact(analysis, provider="bedrock", model="amazon.nova-pro-v1:0"))
suggest_lineage(graph, provider=BedrockProvider(model="amazon.nova-pro-v1:0"))

Tested live on Amazon Bedrock with amazon.nova-lite-v1:0 (explain + enrich). Bedrock per-model output caps are handled automatically (DATAGRAPH_LLM_MAX_TOKENS to override).

Everything else — lineage, relationships, profiling, dimensional modelling, wiki, MCP — needs no LLM at all.

Security

• Deterministic core, no LLM in the loop. Graph, lineage, profiling and the dimensional model are computed from artifacts; an LLM is optional and only explains or suggests (suggestions are schema-validated, must reference existing nodes, are tagged llm and gated by confidence). Nothing an LLM returns is executed.
• Prompt injection. Names, descriptions, docs and SQL are data from your repos and warehouses. Every LLM prompt wraps them in <data> tags with an instruction to never follow instructions found inside; text is stripped of control/bidi characters and truncated; wiki/context output and the MCP server instructions carry the same "untrusted text" notice for downstream assistants.
• Secrets. Connection strings are used only to open a connection; they are never written to the graph, the cache or outputs, and passwords are redacted wherever a DSN is printed. Prefer environment variables / key-pair / SSO auth from your driver.
• Personal data. Profiling keeps counts but masks sample values (min/max/top values) for sensitive-looking columns; --no-top-values disables value sampling; --no-profile skips data access entirely.
• SQL / HTML injection. Identifiers are quoted and literals escaped in every generated query; HTML reports escape embedded JSON.
• MCP server. stdio-only local process (no network port), read-only over the graph file you pass, accepts no connection strings.
• Access. Use a read-only database role; datagraph only issues SELECTs against information_schema and the tables you profile.
• Plugins are Python entry points — install only extractor packages you trust (same trust level as any pip package).

How it compares: Graphify · DataHub · OpenLineage · datagraph

Positioning: OpenLineage is the wire format lineage travels in; DataHub is the catalog it lands in; Graphify is the repo map for an assistant; datagraph is the local data engine that reads your warehouse, dbt and code, and imports OpenLineage / DataHub rather than competing with them. It is deliberately not a catalog (no search UI, glossary, governance or monitoring).

Known limits: code languages are Python and JS/TS (regex-based for JS); call edges are name-resolved (tagged inferred); column lineage needs SQL or a catalog — otherwise a same-name heuristic (tagged inferred) or the opt-in llm fallback applies; dimensional classification is heuristic and always shows its reasons and confidence.

Node ids, provenance, propagation

table:prod.analytics.customer  column:dim_customer.customer_key
dbt:dim_customer               source:raw.customers
file:models/customer.sql       func:src/api.py::customers_endpoint
class:src/models.py::Customer
exposure:revenue_report        job:airflow/load_dim_customer   dag:nightly
task:nightly/build_dim         lambda:GetBookings                api:GET /bookings

Every edge carries a provenance — extracted (from an artifact), inferred (heuristic: name-resolved call, same-name column, name-inferred foreign key) or llm (accepted suggestion); --no-inferred keeps only extracted. Edges are typed and each type knows which way change flows (contains, writes_to, exposes forward; calls, imports, depends_on reverse): impact() walks forward, upstream() backward, lineage() both.

Development

git clone https://github.com/sumit-gupta03/datagraph && cd datagraph
pip install -e ".[dev]"
# 167 tests offline + live Postgres/MySQL in CI (~60 s)
pytest

Docs: docs/TECHNICAL_REFERENCE.md (implementation reference — every module, class and function, with line numbers; also as .pdf/.docx), docs/datagraph-documentation.pdf (how it was built, A to Z) and docs/datagraph-learning-guide.pdf (graphs and lineage from zero). Tagging vX.Y.Z builds wheels, creates a GitHub Release and publishes datagraph-core to PyPI via trusted publishing (.github/workflows/publish.yml).

Roadmap

• Plugin packages for Looker / Tableau / Power BI / Dagster / Prefect / Kafka (the datagraph.extractors entry point is ready)
• Tree-sitter parsers for Java / Scala / Go (today: Python via ast, JS/TS via regex)
• Data-quality rule suggestions from profiles + model (uniqueness of keys, referential integrity, freshness SLAs)
• Incremental per-file rebuilds (today --update skips unchanged inputs)

Authors

Sumit Kumar Gupta and Nitish Pradhan.

License

MIT