DVI — Data Versioning Intelligence

Catch the data incidents that pass every green check.

Quick start · How it works · Signatures · Docs · Contributing · Live demo

DVI is a self-hostable intelligence layer that detects semantic data incidents — the ones where the pipeline runs green but a business number goes silently wrong. It sits on top of your existing stack (dbt, your warehouse, Git); it does not replace any of it. Detection and root-cause ranking are deterministic and explainable — no LLM sits in the decision path.

The problem DVI exists to solve

Data pipelines fail differently from software. A pipeline can execute successfully, keep its schema, meet its freshness SLA, and pass every null/volume check — and still make the business number silently wrong.

-- The whole pipeline stays green. The revenue dashboard is now wrong.
- WHERE status = 'completed'
+ WHERE status = 'COMPLETE'

Before                         After
country = "UK"                 country = "United Kingdom"

Schema unchanged. Row count unchanged. Freshness normal. Yet downstream logic that expects "UK" now under-counts a region, and executive revenue drifts by double digits.

Structural/volumetric observability tools do not catch this class of failure, because nothing structural changed. DVI is built specifically for it.

Quick start

Requires Python 3.11+. DVI isn't on PyPI yet (#13), so install from source:

git clone https://github.com/anuran-de/dvi.git
cd dvi
python -m venv .venv
source .venv/bin/activate        # Windows (Git Bash): source .venv/Scripts/activate
pip install -e ".[dev]"

See it catch a real incident (a deploy silently renames "UK" → "United Kingdom"):

python scripts/demo.py

  Structural checks: schema OK | row_count OK | freshness OK | nulls OK

  DATA INCIDENT
  Severity    : HIGH
  Confidence  : 95% (calibrated, out-of-fold ECE 0.05)

  Suspected data incident from change 'deploy #482 (country normalization)'.
  Value 'UK' (20.0% of the distribution) appears replaced by 'United Kingdom'
  (20.0%) on model.shop.fact_orders; 3 downstream asset(s) affected.

  Evidence:
    * Change 'deploy #482 ...' was deployed 2 min before the first symptom.
    * 'deploy #482 ...' directly changed model.shop.fact_orders, where DVI
      observed: Value 'UK' (20.0%) appears replaced by 'United Kingdom' (20.0%).
    * No corresponding change was observed upstream of the targeted asset(s).

Use it on your own data

DVI runs from a single dvi.toml describing one asset, its before/after snapshots, its dbt lineage, and the changes to attribute against:

asset = "model.shop.fct_orders"
columns = ["revenue", "country"]          # optional; omit = all shared columns

[source]                                  # "file" or "warehouse"
kind = "file"
before = "artifacts/fct_orders.main.parquet"
after  = "artifacts/fct_orders.pr.parquet"

[lineage]
manifest = "target/manifest.json"         # dbt manifest → models + exposures

[[changes]]                               # one or more; RCA attributes to these
id = "pr-1234"
label = "Refactor revenue rollup"
targets = ["model.shop.stg_orders"]
timestamp = 2026-08-30T12:00:00Z

[gate]
fail_on = "high"                          # low | medium | high | critical

dvi analyze --config dvi.toml --output-dir .dvi

It writes .dvi/dvi-report.md + .dvi/dvi-report.json and sets the exit code: 0 (clean or below gate), 1 (gate tripped), 2 (could not run). Full reference: docs/cli.md.

In CI (GitHub Action)

DVI ships a composite Action that runs on a pull request and posts the report as a sticky comment, failing the check when the gate trips:

name: DVI
on: pull_request
permissions:
  contents: read
  pull-requests: write
jobs:
  dvi:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: anuran-de/dvi@main
        with:
          config: dvi.toml

A ready-to-copy workflow ships at .github/workflows/dvi-example.yml.

How it works

DVI turns two snapshots of one asset into an evidence-backed incident (or a clean bill of health):

1. Profiles each column over time — including value distributions, not just counts.
2. Detects change signatures — deterministic tests that recognise the statistical fingerprint of a semantic change (e.g. a dominant category's mass collapsing while a new value absorbs it) without needing to "understand" meaning.
3. Corroborates a detected change against deployments/commits and downstream lineage — an isolated blip stays a symptom; only a change that correlates with a deploy and propagates downstream becomes an incident.
4. Ranks likely root causes with evidence — every claim is backed by observable facts, never a fabricated confidence number.
5. Names the business-level impact — dbt exposures downstream (dashboards, ML features, applications) are named by type, and a business-critical consumer can escalate severity into a critical tier — only for a material change.

Two capabilities make it production-ready at scale:

• Warehouse pushdown profiling — step 1 can run inside the warehouse: a SqlDialect (DuckDB, Snowflake) computes the profile in SQL and only the compact result comes back, so profiling a billion-row table moves a handful of aggregates, not the table. See docs/warehouse-pushdown.md.
• Incident history — detection is stateless by default, but an optional local store gives incidents a stable identity so recurrences dedupe and an asset's history is queryable over time. Opt in with [store] in dvi.toml. See docs/incident-store.md.

Design principles

• Integrate, don't replace. Reuse dbt's lineage, your warehouse, DuckDB. Build only the intelligence layer from scratch.
• Deterministic first. Detection and ranking are deterministic and explainable. An LLM, if used at all, only narrates evidence — it never decides whether something changed.
• Evidence before explanation. Every root-cause claim carries the observable facts that support it.
• Honest confidence. No hand-tuned "92%". Confidence is either omitted (rank
  • evidence) or calibrated and measured on held-out data — a logistic model whose out-of-fold reliability is reported (ECE ≈ 0.05), not asserted.
• Symptom ≠ incident. Corroboration (time × deployment × downstream propagation) is required before anything pages a human.

The signatures

Each is a deterministic test over two column profiles. More specific signatures suppress more general ones on the same column (a rigid ×100 re-encoding reports as unit/scale shift, not a generic distribution shift).

Commodity signatures (null-explosion, cardinality, volume, duplicate-rate, schema/type) are slotted in where cheap.

Why you can trust it

DVI is measured against decoys, not just positives — the hard test for a change detector is staying silent when nothing changed.

python scripts/benchmark.py

  At the shipped default threshold (0.10):
    recall              : 100%
    false-positive rate : 0%

  Operating curve (distribution-shift threshold sweep):
     threshold   recall  fp_rate   false positives
          0.01     100%      43%   3 numeric decoys/negatives fire
          0.08     100%       0%   -
          0.43      80%       0%   distribution-shift positive missed

  Validation on real data (diamonds, 53,940 rows)
  Real-vs-real false positives: 0/210 column-checks fire (0%) across 30 disjoint splits
  Injected-rename recall       : 30/30 (100%)

  Calibrated confidence (per-symptom, k-fold cross-validated)
  Dataset: 58 fired symptoms, 34 real (59% positive)
  Out-of-fold ECE: 0.0466   MCE: 0.2165   Brier: 0.0051

Documentation

Live demo: the operator UI is deployed at dvintelligence.vercel.app — an editorial landing page plus an incident dashboard, detail timeline, and blast-radius graph, all rendered from real pipeline output.

Roadmap

DVI is built as a walking skeleton — the riskiest, most novel part (does semantic detection + causal ranking actually work?) is proven first; UI and connectors come last. Every milestone below is complete and green in CI; see the CHANGELOG for the detail.

Explicitly not built yet

• PyPI package (#13) — install from source until then.
• Automatic BI/ML lineage discovery (Tableau/Looker/feature stores) — downstream assets register via dbt exposures until then.
• Warehouses beyond DuckDB (executed in CI) and Snowflake (dialect + SQL-gen tests, not CI-executed) — another warehouse needs a new SqlDialect.
• Auto-derived change events — [[changes]] is declared explicitly in dvi.toml; DVI does not yet infer changes from commit/deploy history.
• Multi-asset runs — one dvi analyze run covers one asset.
• Forges beyond GitHub and any autonomous remediation.

Contributing

Contributions are welcome — DVI is early and the open issues are the best place to start.

pip install -e ".[dev]"
pytest              # the full suite (also runs the demo + benchmark end to end)
ruff check .        # lint

A few conventions this project holds to:

• Test-driven. New behavior lands as a failing test first, then the code to pass it. CI runs the suite twice under different hash seeds as a determinism guard, so avoid depending on set/dict iteration order.
• Deterministic and explainable. Detection and ranking must not depend on an LLM or on unseeded randomness. Confidence numbers are either omitted or measured — never hand-tuned.
• Few dependencies on purpose. The engine uses only polars, duckdb, networkx, and pydantic. New runtime dependencies need a strong reason (this is why, e.g., Snowflake's pyarrow-pulling driver isn't in CI).

Open an issue to discuss anything larger before you build it.

License

MIT — see LICENSE.