runreceipt

A test run that reports 100% over three of ten tests is green in every CI you own. runreceipt pins how many items a run is supposed to cover, then recomputes the number afterwards and says out loud what went missing.

Zero dependencies. It is the thing that checks the other tools, so it imports none of them.

$ runreceipt fix --tool langsmith --inputs data/questions.jsonl -o run.lock
pinned 10 items from 1 input file(s); tool langsmith==0.10.18 (installed)
lock   run.lock  sha256 68d1e39f23d9a54a9d239ee1e0c796ec

$ pytest --junitxml=results.xml          # something silently drops 7 items
$ runreceipt verify results.xml --lock run.lock
runreceipt  RED
artefact    results.xml  (adapter: junit)

  pinned before run   10
  present in artefact 3
  reached a verdict   3   (passed 3, failed 0)
  lost                0
  skipped             0
  honest denominator  3

  reported pass_rate    1.0000   over denominator 3
  recomputed           1.0000   over denominator 3

  [RED] SUITE_SHRANK
      10 items were pinned before the run, 3 appear in the artefact, and the
      run surfaces no loss; the missing 7 are invisible to anyone reading the
      figure

  instrument   7 rules, each probed on a known-bad and a known-good input
  expires      langsmith==0.10.18 over input 92099c6974199bac
  receipt      sha256 54e156047a740dd5072477254c2ccc0f

$ echo $?
2

Note what the reported and recomputed figures do here: they agree. Both say 1.0000. Recomputing the artefact against itself can never find this class of defect — the artefact no longer contains the missing items. Only the number pinned before the run does. That is why fix exists.

The three properties

Every receipt carries what the artefacts we examined did not:

1. The denominator is stated, and it was fixed before the run. Not derived from whatever survived.
2. Losses are named as losses. An item that raised is a non-pass, not an absent question. It stays in the denominator.
3. The green has an expiry. A receipt is valid for one release over one input set: runreceipt check --lock run.lock re-derives both and prints VALID or EXPIRED. A verdict with no expiry is a verdict about the past.

$ runreceipt check --lock run.lock
VALID    pinned at 2026-08-14T20:17:20Z; langsmith==0.10.18 and input 92099c697419 are unchanged

$ echo '{"id": "q10", "question": "one more case"}' >> data/questions.jsonl
$ runreceipt check --lock run.lock
EXPIRED  a result derived under this lock no longer applies:
  - the input set changed (pinned 92099c697419, now 87b22cfe46d5)
$ echo $?
2

Install

pip install runreceipt

Commands

Exit codes: 0 consistent · 2 findings · 3 artefact unreadable · 4 the instrument failed its own probe. There is no code that means probably fine. An artefact that cannot be parsed is RED-BLIND, never a pass.

fix takes nothing on trust. The item count is counted from the input set rather than read from a field, and the tool version is resolved from the installed distribution rather than from what you typed — if those disagree with your claim, the receipt says so. A pinned number that was never measured would just move the defect one file earlier.

What it checks

The regression set is real releases

Each fixture reproduces a defect found in a shipped version of a tool people run in CI today, and carries a _reproduces field naming it. selftest replays all of them on every commit:

$ runreceipt selftest
  promptfoo-zero-tests       expect RED   got RED       ok  EMPTY_DENOMINATOR
  deepeval-judge-raised      expect RED   got RED       ok  VERDICT_CONTRADICTS_ERROR
  langsmith-3-of-10          expect RED   got RED       ok  SUITE_SHRANK
  weave-mean-survivors       expect RED   got RED       ok  LOSS_UNREPORTED,DENOMINATOR_OVER_SURVIVORS,FIGURE_MISMATCH
  galileo-buffer-counter     expect RED   got RED       ok  COUNTER_MODULO_BUFFER
  langfuse-8-of-100          expect RED   got RED       ok  LOSS_UNREPORTED,DENOMINATOR_OVER_SURVIVORS,FIGURE_MISMATCH
  swebench-dropped-row       expect RED   got RED       ok  SUITE_SHRANK
  honest-all-pass            expect GREEN got GREEN     ok
  honest-some-fail           expect GREEN got GREEN     ok
  honest-losses-named        expect GREEN got GREEN     ok
  honest-skips-excluded      expect GREEN got GREEN     ok
  unreadable-artefact        expect BLIND got BLIND     ok
SELFTEST OK  7 rules probed both ways, 12 fixtures verified

The four honest-* fixtures are the half that matters. honest-losses-named has two items that raised and reports lost: 2 — it must stay green. Errors are not the defect; hiding them is. A detector that reddens on any imperfect run is a constant, and a constant detects nothing.

Why the rules probe themselves

Each rule ships two inputs: one it must fire on and one it must stay silent on. If any rule fails either probe, the whole receipt goes RED and the exit code is 4 — even when no finding fired. Silence from a broken instrument is not a pass, and that is precisely the failure being sold against here: every defect in the table above is a green produced by a number nobody could recompute.

The test suite applies the same standard to itself. One test disables every rule and asserts the seven defects then stop being detected — if they still came out RED, the RED came from somewhere other than the rules and the regression set proves nothing. Another asserts that the same neutered build still refuses to print GREEN, and exits 4 rather than 2, so "nothing found" and "nothing looked" never collapse into the same signal.

Adapters

native · junit (pytest, and anything that writes JUnit XML) · promptfoo · deepeval · weave · galileo · langfuse · swebench. Sniffed automatically; override with --adapter.

Adding a vendor costs one adapter and zero rules: adapters lower an artefact into a normalized Run, and the rules only ever see a Run.

What this tool does not claim

• Not that any tool's authors acted in bad faith.
• Not that any number was falsified. Every defect above is an ordinary bug — a bare except, a buffer remainder, a mean over a filtered list.
• Not that a green receipt means the model is good. It means the figure survives recomputation against a denominator fixed before the run.
• Out of scope: how a leaderboard ranks or badges rows. runreceipt reads one run's artefact and says whether its number holds.

License

MIT.