Plumb

Execution-grounded research-integrity verifier: re-runs a paper's own artifacts on your compute and returns a per-claim, reproducible verdict.

Point Plumb at a paper and the code behind it. It pins the code, builds an environment, re-runs the artifacts, re-derives the paper's quantitative claims from what the run actually produced, and renders an honest verdict per claim — REPRODUCED / WITHIN-TOLERANCE / DIVERGED / UNVERIFIED — decided by re-execution, never by a model's opinion of the paper.

Quickstart · How it works · The verdict · The number so far · Vision · Roadmap · Contributing

_{The card is a real replay, not a mock. It shows the committed record of the first notebook-computed paper through the spine (fixtures/gate/rcai). The numbers the paper printed in its own notebook cells were re-derived by re-executing that notebook and bound through notebook_cell locators. C1 conformance on this paper is 0/31 — reported as conformance, never as coverage.}

Why Plumb

Three kinds of tools sit near the literature. Text-mining tools extract numbers. LLM judges opine on them. Plumb is the third thing: the verifier. It answers the question none of the others do — "does the claim hold when you actually run it?" — by re-executing the paper's own artifacts and diffing the re-derived value against the reported one.

Why that question matters, and why a judge can't answer it:

• Frontier models hit only ~6.1% precision / ~21.1% recall on real errata-worthy errors (SPOT, arXiv 2505.11855) — well below the bar for flagging a paper.
• Only ~3.2% of published computational notebooks reproduce (widely cited) — a paper's numbers often cannot be rebuilt at all, which is itself the finding.
• Free AI tools cannot reliably flag even retracted literature (JMIR 2026) — a guess about correctness is not verification of it.

Plumb's verdict is grounded in execution, not opinion — which means it gets better as base models improve (they write cleaner claims and better bindings), never redundant.

• 🪢 The name. A plumb line is the oldest tool for testing whether something stands true — and "to plumb" is to investigate to the bottom. Plumb asks of a paper: does the claim hold when you actually run it?
• 🔁 Re-derive, don't re-read. The value is re-derived from the paper's own run on your compute — never from a committed output, never from a stale file (a result that predates the run is UNVERIFIED, never parsed).
• ⚖️ An honest verdict contract. REPRODUCED means the paper's number was re-derived within its written precision; DIVERGED means the artifact's own run contradicts the paper's claim — and UNVERIFIED is never dressed up as either. A discrepancy that could be our harness's fault resolves to UNVERIFIED, never DIVERGED.
• 🧾 Signed, replayable bundles. Every finding ships with a hash-listed, ssh-keygen -Y-signed record — paper hash, tree hash, frozen environment, traces, verdicts — that a third party can replay byte-identical. A finding nobody can replay is an opinion.
• 📈 A compounding discrepancy corpus. Every (claim, re-derived value, verdict) banks as a labeled, replay-validated case; plumb corpus report pools coverage / precision / recall with denominators and label-authority markers, measured against human labels.
• 🔒 Runs on your compute. The paper, the repo, and the data never leave your machine; only hashes and the verdicts you choose to publish ever do.
• 🧪 Test-first, deterministic. A 2,000+-test offline suite pins determinism byte-for-byte; no network is reachable from any test; one pinned runtime dependency (pypdf).

Honesty is the whole product. UNVERIFIED is never rendered as REPRODUCED, extraction conformance is never reported as coverage, and a DIVERGED that owner-review refutes (a run-to-run artifact, not a paper error) stays refuted — the corpus says so. Read The number so far before trusting any claim.

Quickstart

Install (PyPI plumb-verify):

uvx --from plumb-verify plumb --help   # run without installing
pipx install plumb-verify              # or put the `plumb` script on your PATH

The import package and the console script are both plumb; the distribution is plumb-verify because the PyPI name plumb belongs to an unrelated, older library.

The 60-second demo — replay a committed record, byte-identical. No network, no environment build, no API key: the paper comes with a record of its own run, and the verdicts are re-derived from those committed bytes alone.

git clone https://github.com/haqaliz/plumb && cd plumb
uv run plumb verify --from-record fixtures/gate/rcai

31 claims, 30 bound, 30 REPRODUCED, 1 UNVERIFIED, 0 DIVERGED — the first notebook-computed paper through the spine, and the replay is byte-identical every time. Add --json for the canonical document, or --bank to fold the case into the discrepancy corpus.

Verify a paper of your own, live:

uv run plumb verify <paper> <repo> [--rev REV] [--bindings bindings.json]
#   <paper>  a local PDF/Markdown — the claims are extracted and selected by rule
#   <repo>   a local dir, git URL (with --rev), or archive — pinned checkout + env
#   --bindings   your user-written bindings: which captured value each claim is checked against

Live mode is expert-mode today. --bindings is required and hand-written; there is no binding proposer yet. That gap is the next capability, not a footnote — see docs/LAUNCH.md.

Develop:

uv sync && uv run pytest        # 2,080 tests, no network, one runtime dep (pypdf)

Exit contract: 0 iff every claim is decided; 1 on any UNVERIFIED or named cause; 2 on usage. Never a traceback; a harness failure is never a DIVERGED.

How it works

The spine: intake → run → re-derive → verdict → bundle (plumb verify runs it end to end).

paper (text/PDF)                      repo/data (path | git URL --rev | archive)
       │                                        │
       ▼                                        ▼
 C1  extract & select claims   ──┐      C2  pinned checkout + frozen environment
       │   (typed Claim records) │          │
       │                         │          ▼
       │                         │  C3  run the repo's own entry point
       │                         │      (script | [project.scripts] | root main.py
       │                         │       | notebook via nbconvert — freshness-guarded,
       │                         │        content-addressed outputs)
       │                         │          │
       └──────────────┬──────────┘          ▼
                      ▼                re-derived values
                 C4  bind each claim to a value via a locator
                     (JSON pointer · stdout regex · CSV cell ·
                      notebook cell output · notebook HTML table cell)
                     → compare within the paper's written precision or a declared tolerance
                     → REPRODUCED / WITHIN-TOLERANCE / DIVERGED / UNVERIFIED
                      │
                      ├────────────►  C5  discrepancy corpus (bank every case)
                      ▼
                 C6  signed, replayable bundle (paper hash + tree hash + env + traces + verdicts)

• The claim is the unit of work. C1 emits typed, self-describing Claim records — reported value (verbatim), units, metric, paper location — through a deterministic admission gate; a candidate the paper didn't make is recorded as a non-claim with a named cause, never a silent drop. An optional BYOK LLM proposer may suggest candidates; every proposal is re-grounded against the paper text before admission and never assigns a verdict.
• Reproduce what ran, not what was committed. C3 executes the repo's own entry point in a working copy under the run area and captures stdout / JSON / CSV / notebook cells into a per-run object store keyed by SHA-256. An output whose mtime predates the run start is STALE_ARTIFACT — its bytes are never read.
• Notebooks are first-class. A root *.ipynb is the fallback entry point (jupyter nbconvert --execute); each output-bearing cell becomes a content-addressed artifact, and a claim binds into it — including <td> cells inside pandas Styler HTML, the way notebooks actually render tables.
• The verdict is decided by code, in pinned Decimal. No float reaches a comparison; run causes govern every claim first; every DIVERGED carries review_required.

The verdict

A model may extract a claim or propose a binding; execution assigns the verdict.

The number so far

Three real papers are banked — each record replays byte-identical and every labeled verdict is owner-reviewed:

• AgroDesign (arXiv:2603.09041) — 86 rule-defined claims, 85 REPRODUCED, 1 DIVERGED: a Shapiro-Wilk p reported as 0.034 that its own code computes as 0.03455, confirmed genuine on review and unchanged in a paper-era environment. C1 recovered 86/86 of its claims (conformance to the curated rule; fixtures/gate/agrodesign/README.md).
• Perrin (arXiv:2401.11842) — 16 claims, 2 REPRODUCED, 14 DIVERGED … refuted on review: the artifact is not run-to-run reproducible at the paper's written precision (unseeded worker RNG; noise ~0.005–0.010 vs written precision 0.001; four verdict flips across three runs). A reproducibility finding about the artifact, not a paper error — recorded as refuted, never published as a discrepancy.
• Burtsev (arXiv:2609.00137) — the first notebook-computed paper through the spine: 31 claims, 30 REPRODUCED through notebook_cell locators, 1 UNVERIFIED (NO_BINDING), 0 DIVERGED; 70 s wall; a second full run changes zero verdicts (fixtures/gate/rcai/README.md).

Totals: 133 claims, 132 bound, 117 REPRODUCED, 15 DIVERGED (14 refuted, 1 confirmed), 1 UNVERIFIED — precision/recall 1/15 against owner labels. The 1/15 is the honest number: one republishable discrepancy out of fifteen flagged, because fourteen were the artifact's own run-to-run noise. The runnable-paper space is thin (consistent with ~3.2% notebook reproduction), and growing the panel is blocked by named engine gaps — figure-with-data locators and the fact that most papers ship no code at all (the C7 no-code path). The C2 poetry env policy gap is now closed (src/plumb/intake/env.py: a poetry-locked checkout is classified "poetry" and built via uv pip install, and a build that installs nothing is a named failure, never a silent empty venv; fixtures/gate/perrin/README.md, probe note 2026-10-17).

How to read that number: the denominators, the label authority, the reason fourteen flags were refuted, and the answer to "so your tool was wrong 14 out of 15 times?" are in docs/METHOD.md. What a DIVERGED does and does not assert — and the right-of-reply process — is in docs/ETHICS.md.

Docs

• VISION.md — the thesis, the moat, non-goals.
• CLAUDE.md — guardrails and the verdict contract.
• docs/ROADMAP.md — phased plan and risk register (R1–R7).
• docs/LAUNCH.md — the ordered punch-list to a public launch (Product Hunt).
• docs/METHOD.md — the panel, the verbatim corpus report, and how to read 1/15.
• docs/ETHICS.md — what a verdict asserts, publication rules, right of reply.
• docs/technical/CAPABILITY_ROADMAP.md — C1..C8 capabilities in build order.
• docs/technical/ARCHITECTURE.md — the design.
• CHANGELOG.md — release history.
• fixtures/gate/ — the banked real-paper records with provenance.

Contributing

Test-first is the house rule: every capability lands with its failing test written first, the suite runs offline, no network is reachable from any test, and a verdict is never assigned by a model. See CLAUDE.md before opening an issue.

Support the project

If this project is useful to you, you can chip in with a crypto donation:

Lineage

Plumb generalizes a "reproduce & verify published work" capability out of genomics and points it at the literature at large — the deterministic spine (intake → run → re-derive → verdict → bundle) plus a compounding discrepancy corpus that gets sharper as base models improve.