veridia 0.1.0

Veridia - a nutrition label for data: every computed number gets an ancestry.

Pedigree

A nutrition label for data. Pedigree records the ancestry of every computed number — which rows, files, code, and versions produced it — and lets anyone ask a number why it is what it is. Answers are verified by actually re-running the recorded steps, and every answer carries an honest confidence label. It is open, local-first infrastructure: Apache-2.0, SQLite, SHA-256 — no server, no account, no network, no telemetry.

The package name is veridia (the import and CLI stay pedigree): pip install "veridia[xlsx]" — docs/install.md has the extras and an install-from-checkout form.

Sixty seconds: find the number somebody typed in

A Q2 revenue workbook. Four regional totals roll up to $985,000. Three are real =SUM(...) formulas; one was typed in by hand — padded $95K over its actual deals. On the screen they are indistinguishable. Three commands:

pedigree ingest xlsx demo/excel_audit/revenue_review.xlsx
pedigree why "xlsx:demo/excel_audit/revenue_review.xlsx#Summary!B6"
pedigree why "xlsx:demo/excel_audit/revenue_review.xlsx#Summary!B5"

value: 985000
[Recorded]

xlsx.formula  -> .../Summary!B6  [Recorded]
  xlsx.formula  -> .../Summary!B2  [Recorded]     <- North: calculated
    source: .../Detail!C2:C3  [Source]
  xlsx.formula  -> .../Summary!B3  [Recorded]     <- South: calculated
    source: .../Detail!C4:C5  [Source]
  xlsx.formula  -> .../Summary!B4  [Recorded]     <- East: calculated
    source: .../Detail!C6:C8  [Source]
  source: .../Summary!B5  [Source]                <- West: a dead end

no formula cell Summary!B5 recorded in revenue_review.xlsx

West has no formula behind it. Somebody typed it. Pedigree found the one hand-entered number in the report — the classic hardcoded "plug" — in one click. The full walkthrough is demo/excel_audit/SHOW.md; the 15-minute meeting version is docs/demo-script.md.

The honesty rule

Every answer is labeled, and the label is load-bearing:

• Exact — derived by verified replay: every step re-run (twice), fingerprints checked against the recorded run.
• Recorded — step-level truth from receipts; no row-level claim.
• Opaque — a black-box step (a UDF, an unknown frame, unanalyzable SQL). The chain stays connected and labeled, never invented.

Pedigree never silently guesses. If a replay can't be verified it says so and degrades honestly — a confident wrong ancestry is treated as the worst possible bug in the project.

What it is not

Read this before the feature list — same words as the honest comparison:

• Not a catalog. No ownership, discovery, glossary, or governance workflows. If your shop runs DataHub or Collibra, Pedigree complements them — it answers a different question. The word is and, never instead.
• Coverage is narrow by design. Live capture for pandas, DuckDB, and Postgres; ingestion for Excel workbooks, dbt runs, and declared-IO Python tasks. No Snowflake, Spark, SQL Server, or BI tools — and no promised dates for any of them.
• Opaque is common in real pipelines. First contact with a messy codebase will show Opaque steps. That label is the tool working.
• Single-machine. No shared or team ledger. The sharing mechanism is the evidence pack (pedigree export) — one self-contained HTML file.

How it works

Record cheaply always; reconstruct exactly, only when asked.

1. Receipts (always on, near-free). Every computation step emits a tiny content-addressed record naming its inputs, code, and outputs by fingerprint — never copying data. Measured overhead on the benchmark pipeline: ~3.5% median (budget: 5%, enforced by a test gate forever; see docs/limits.md for measured numbers at scale).
2. Replay (on demand, exact). When you ask about a number, the resolver walks the receipt chain backward and re-runs only the steps that matter against the recorded input versions, deriving row-level detail. Expensive, but paid at ask time and cached.

Chains cross process and system boundaries by content fingerprint handshake: a file written by pandas and read by DuckDB has the same fingerprint on both receipts, so pandas -> file -> SQL -> report traces as one chain. Nothing is stitched by filename, timestamp, or guesswork.

What works today (v0.1)

• Taps (add import pedigree to a script — that is the only change): pandas (read/merge/filter/groupby/reduce/write), DuckDB and PostgreSQL (inputs discovered engine-side, never by regexing your SQL; probes can never alter your results or transactions).
• Ingesters (for systems that aren't a Python process): Excel (pedigree ingest xlsx, cell-level ancestry from the formula graph), dbt (pedigree ingest dbt after a run), Airflow-style tasks (a pedigree_task decorator), and pedigree run -- <cmd> to wrap any command as one receipt.
• Questions: why (step- and row-level ancestry, per-row influence), verify (drift check), replay (exact re-execution), diff (why a number changed between runs), export (a self-contained HTML evidence pack for workpapers), serve (local viewer on 127.0.0.1).
• Audit-grade history: the ledger is hash-chained (pedigree audit detects removed or edited history and names the first bad entry), with opt-in Ed25519 receipt signing. Receipts are append-only forever.
• A public standard: the receipt format is a frozen, versioned spec (spec/receipt-v1.md + JSON Schema) with a conformance validator (pedigree validate) and an adapter SDK — anyone can write a tap.

Install

See docs/install.md — pip/pipx, optional extras ([duckdb], [postgres], [xlsx], [sign]), and a section for IT departments. Python 3.11+. From a checkout: pip install -e ".[xlsx]".

Then try it in 60 seconds (the golden two-process demo):

python demo/generate_data.py
python demo/script_a.py                  # pandas: clean + join -> parquet
python demo/script_b.py                  # pandas: aggregate -> report.json
pedigree why report:total_revenue        # the whole chain, across processes
pedigree why report:total_revenue --rows # the 1000 contributing rows [Exact]
pedigree audit                           # the history is tamper-evident

Supported platforms

The support matrix is exactly the set of combinations CI runs and nothing more (docs/install.md keeps the authoritative copy):

OS	Python	pandas	Proven by
Windows, Ubuntu, macOS	3.11–3.14	current 3.x	fast suite green per OS x version
Ubuntu	3.11	2.0.x floor (numpy<2)	fast suite green
Ubuntu	3.12	current 3.x	fast suite + live PostgreSQL 16

CI also proves the demo's artifact fingerprints are byte-identical across all three OSes on every push.

Documentation

• Install · Quickstart (10 min) · Reading a Pedigree answer (start here if you're not a programmer)
• CLI reference · FAQ · Limits (measured) · Operations
• Security & compliance: Security whitepaper (every claim linked to a test or measurement) · Compliance mapping (not legal advice — says so up front)
• Making the case internally: Honest comparison · Two-week pilot playbook · 15-minute demo script
• Extending: Writing a tap · Support policy

Honest limitations

• Row-level answers stop at opaque steps (UDFs, in-place mutations); past them you get Recorded with a plain-language reason, not a guess.
• Large intermediates use sampled fingerprints (honest-but-weaker, and marked as such); SQL row-level tracing is step-level only.
• Macro-enabled workbooks are refused (a macro can set any cell); volatile and dynamic Excel constructs degrade honestly to Opaque.
• Signing key files cannot be POSIX-locked on Windows (documented caveat).
• The full register, each with its measured number or test: docs/limits.md and docs/security.md §9.

Sustainability & the exit story

From the support policy, verbatim:

Pedigree is maintained by one person. If maintenance stops, adopters lose nothing they cannot keep: the receipt spec is frozen, public, and versioned (spec/receipt-v1.md + JSON Schema); the ledger is a plain SQLite file of canonical JSON on your own disk; the conformance validator and SDK are Apache-2.0; there is no server, no account, and no network dependency that can be turned off. Any competent engineer can read a ledger with sqlite3 and the schema in an afternoon. The exit is: copy your file.

Security reports: see SECURITY.md.