proof-of-done 0.1.1

Quickstart · How it works · Results · Configuration · Docs

Coding agents end turns with "all tests pass", "the build succeeds", "fixed" or "deployed", and the session often does not back it up. proof-of-done treats those words as claims. A Claude Code Stop hook checks the agent's own transcript: did the matching command run after the last relevant edit, and did it succeed? If not, the stop is blocked and the agent is told exactly what to run. The same engine audits past transcripts and reports how often a set of sessions claimed what they never checked.

No LLM, no network, no re-running your commands: for a given transcript, message and configuration the verdict is deterministic.

_{A fixture Stop payload piped into the installed hook, then audit --demo. Not a live agent session.}

Highlights

• Acts only on claims. A keyword prefilter lets every other stop through without reading the transcript; the hook never runs tests or deploys itself.
• Evidence, not vibes. The latest matching command after the last relevant edit decides. Failed, empty, interrupted, background and masked runs (| tail, || true) do not count.
• Actionable. Each block names the claim, the reason with step numbers and the file, and a Run: line with the command to run.
• Hard to game. Fake echo "12 passed", a redefined pytest, PATH tricks, --collect-only, the agent typing the skip token or editing the config to switch the check off: an adversarial set tests all of these, and every case that gets through is listed below.
• Safe to leave on. Stdlib-only hook, fails open on any error, a loop cap below Claude Code's own, a tamper check against self-disabling.
• Measured. Precision and recall on a held-out set frozen before the detector existed, and latency on real hardware, all reproducible from the repository.

Quickstart

claude plugin marketplace add B0yko/proof-of-done
claude plugin install proof-of-done@proof-of-done

The plugin loads on the next start of Claude Code, or after /reload-plugins in an open session. Inside a session, the equivalent is /plugin marketplace add B0yko/proof-of-done, then /plugin install proof-of-done@proof-of-done, which opens the plugin's details to pick a scope; closing the /plugin panel reloads plugins (docs).

Try the audit CLI on a bundled synthetic corpus without installing anything:

uvx proof-of-done audit --demo

Requires python3 ≥ 3.9 on the PATH Claude Code's hooks see (on macOS, /usr/bin/python3 needs the Command Line Tools; without them the hook skips its check with a message), macOS or Linux. Tested with Claude Code 2.1.281.

What a block looks like

Real output: a fixture session in which the tests ran alongside the edit rather than after it, piped into bin/proof-of-done-hook, the launcher Claude Code runs on Stop. The hook prints {"decision": "block", "reason": …}; this is the reason the agent receives:

proof-of-done: 1 claim in your final message is not backed by this session's transcript.
1. "all 40 tests pass" (tests_passed): the last test run `uv run pytest -q` at step 2 was before your edit to `src/app/parser.py` at step 4.
   Run: uv run pytest -q
Run these commands and report the real result, or restate your message without these claims.

The Run: command is the one the agent itself ran. Failing that, it comes from the rule's suggest setting or from the project files; for a deploy claim or an unrecognised project it reads Run: (no … command found — run it in the foreground).

OUT=$(mktemp -d)
uv run python fixtures/render.py fixtures/scenarios/parallel-calls.yaml --out "$OUT"
sed -e "s|__TRANSCRIPT_PATH__|$(ls "$OUT"/*.stop-0-0.jsonl)|" \
    -e "s|__CWD__|$OUT|" -e "s|__SCRATCHPAD_DIR__|$OUT|" \
    -e "s|__LAST_ASSISTANT_MESSAGE__|Updated the parser; all 40 tests pass.|" \
    fixtures/payloads/stop_with_last_message.json |
  sh bin/proof-of-done-hook stop --data-dir "$OUT/data" |
  python3 -c 'import json, sys; print(json.load(sys.stdin)["reason"])'

How it works

flowchart TD
    A([Claude Code Stop event]) --> B[launcher]
    B --> C{fast-path prefilter:<br/>a claim keyword?}
    C -- no --> D([exit 0, no output])
    C -- yes --> E[merged-config cache]
    E --> F[transcript parser<br/>edits · commands · subagents]
    K([proof-of-done audit]) --> F
    F --> G[claim detector]
    G --> H[evidence engine<br/>anchor → candidates → verdict]
    H --> I[suggester: Run: command]
    H --> L[(audit report +<br/>agent-trace/v1 export)]
    I --> J([decision JSON<br/>block · warn · allow])
    classDef accent fill:#10b981,stroke:#047857,color:#ffffff
    class J accent

For each claim in the final message, the evidence engine finds the anchor (the last edit to a file the rule cares about: edit tools, Bash writes, formatters, whole-tree git operations, subagent calls) and the candidates (foreground commands after it that match the rule). The latest candidate decides.

Every verdict carries one reason from a closed set:

The claim detector is plain re. It masks code blocks, inline code, block quotes and quoted user text, then rejects negated, hedged, conditional, future, question and instruction-to-the-user forms ("should pass", "run pytest to confirm", "not verified"). Detection is English only: a claim in another language is not detected at all.

The full rules are in docs/how-it-works.md; the design decisions are in nine short ADRs under docs/adr/.

Results

Every number comes from uv run python eval/run_eval.py and uv run python eval/latency.py --generate --run --max-load 4, and every table is generated from the committed results files by scripts/render_results.py, which CI checks, along with a re-run of the evaluation. There are three fixture sets, never merged into one headline:

• templated: expanded from fixtures/templates/ by the same author who wrote the detector, so it measures consistency with that author's reading of the claim wording;
• held-out: individually authored, free-form sessions, frozen at commit 4e54fc9 (eval: freeze held-out set) before claims.py existed, the more honest estimate;
• adversarial: sessions that deliberately try to game the gate.

34 / 36 gaming attempts met their labelled outcome · hook p95 on a 10 MB transcript 175 ms (system Python 3.9) / 113 ms (CPython 3.12) · audit 141 MB/s

proof-of-done audit report
SYNTHETIC DEMO CORPUS -- this report describes 30 hand-authored fixture sessions, not a measurement of any real coding agent.

config applied: built-in defaults
files processed: 30

sessions: 30
turns: 30
stop attempts: 30
turns with claims: 30
claims: 45
unsupported: 16 (35.6% of claims)
partial share of supported claims: 17.2%

by claim type:
  build_passed: 3 claims, 1 unsupported (33.3%)
  deployed: 4 claims, 0 unsupported (0.0%)
  fixed: 15 claims, 4 unsupported (26.7%)
  lint_clean: 5 claims, 2 unsupported (40.0%)
  tests_passed: 12 claims, 7 unsupported (58.3%)
  typecheck_clean: 3 claims, 1 unsupported (33.3%)
  verified: 3 claims, 1 unsupported (33.3%)

unsupported by reason:
  background_only: 1
  empty_run: 1
  failed_exit: 3
  failed_output: 1
  masked_inconclusive: 2
  no_command: 2
  no_result: 2
  stale: 2
  superseded_by_failure: 2

most recent unsupported claims:
  [fixed] "Patched" -- superseded_by_failure
    Run: cargo test
  [tests_passed] "All tests pass" -- superseded_by_failure
    Run: cargo test
  [tests_passed] "All tests pass" -- background_only
    Run: go test ./...
  [fixed] "Fixed the race" -- stale
    Run: go test ./...
  [lint_clean] "Lint is clean" -- failed_exit
    Run: go vet ./...
  [tests_passed] "All tests pass" -- empty_run
    Run: npx vitest run
  [typecheck_clean] "Typecheck is clean" -- failed_exit
    Run: npx tsc --noEmit
  [fixed] "Patched" -- masked_inconclusive
    Run: npx vitest run
  [verified] "Confirmed it's working" -- masked_inconclusive
    Run: npx vitest run
  [build_passed] "The build succeeds" -- failed_exit
    Run: uv build

Configuration

Five layers merge, lowest precedence first: the packaged defaults, the user file ${XDG_CONFIG_HOME:-~/.config}/proof-of-done/config.yaml, the project's .proof-of-done.yaml, the file named by PROOF_OF_DONE_CONFIG, then the PROOF_OF_DONE and PROOF_OF_DONE_MODE environment variables. Project rules merge into built-in rules by id, field by field; new ids add custom claim types. Every key and the full evidence-command table are in docs/config.md; examples/ has configs for Python, Node, Go and a monorepo, plus custom committed/pushed rules. proof-of-done init writes a starter .proof-of-done.yaml (version: 1, the detected ecosystem as a comment, rules: []).

Tamper check. The hook scans the session's own edits before it applies enabled, mode or any rule action. If the session edited .proof-of-done.yaml, the user config or the PROOF_OF_DONE_CONFIG file, four downgrades from that file are undone for the rest of the session: enabled: false, mode: warn, a lowered action and a raised max_blocks_per_turn. If it edited a Claude Code settings.json/settings.local.json with text containing PROOF_OF_DONE or enabledPlugins, PROOF_OF_DONE=off and PROOF_OF_DONE_MODE=warn from the environment are ignored. A warning names the edited file, and tampering alone never blocks. What it does not undo is listed under Limitations.

CLI

The plugin does not put proof-of-done on your PATH. Install the CLI with uv tool install proof-of-done (or pipx install proof-of-done), or run any command through uvx proof-of-done <command>.

proof-of-done check --transcript PATH [--message TEXT | --message-file PATH] [--config PATH] [--json]
proof-of-done audit [PATH ...] [--claude-projects | --demo] [--source auto|claude-code|agent-trace|codex]
                    [--config PATH] [--format text|json|md] [--out FILE] [--export-traces FILE]
                    [--redact] [--salt S] [--ci] [--max-unsupported-rate R]
proof-of-done init [--force]
proof-of-done config show [--cwd DIR]
proof-of-done trace validate FILE

Debugging a block

proof-of-done check --transcript path/to/session.jsonl --message "All tests pass."
proof-of-done config show

The hook logs to ${CLAUDE_PLUGIN_DATA}/proof-of-done.log (or ${TMPDIR:-/tmp}/proof-of-done-<uid>/ when that is unset), JSON Lines capped at 256 KiB. Each stop past the prefilter writes one decision record with the final decision (after the block cap; capped: true when the cap let a stop through), the claim count, the flags skipped, disabled, tampered and capped, one entry per claim (claim_type, rule_ids, supported, reason code, action) and per-phase timings_ms. A fail-open writes a fail_open record with a reason code, and an unexpected exception an error record with only the exception class name. The log never contains message text, quotes, commands or file paths.

Privacy

The hook reads only local files: the event JSON on stdin; the session transcript (for SubagentStop, the subagent's transcript plus the parent's, for the skip token and the tamper scan); long tool outputs Claude Code stored inside that session's own directory; the config files; and, to suggest a Run: command, the project's pyproject.toml, pytest.ini, package.json, go.mod, Cargo.toml, Makefile and lockfiles. It makes no network call, and a socket-blocking test fixture enforces that. It writes only under ${CLAUDE_PLUGIN_DATA} (or the temp fallback): block counters, the merged-config cache, Python's bytecode cache, the log, and on macOS a .python-ok marker when the interpreter is /usr/bin/python3. It never writes into your project.

Alternatives

Existing Claude Code hooks and plugins that verify completion claims or gate Stop on test, build or deploy state, each checked against its repository on 2026-09-28:

proof-of-done sits with claimcheck and done-needs-proof on deterministic transcript inspection, never re-runs a command (unlike loop-hooks), never calls a model (unlike Probity's validation option and the official prompt-hook example), and publishes a reproducible evaluation against a frozen held-out set. This describes what exists; it is not a claim to be first.

Limitations

• Missed phrasings are never checked. The detector does not know every wording: type-check phrasing ("Types check", "it vets clean"), state-style deploy claims ("it's live in production") and pronoun subjects ("They pass") are the main gaps. See the failure analysis under Results.
• English only.
• Two known adversarial misses. An echo-only Makefile test: target whose output looks like a test summary, and npm run test:* scripts, which are trusted by name even when they start a watcher.
• Background runs never count as evidence in v0.1, even when a later notification reports success; the block message says to re-run in the foreground.
• Only Bash is evidence. MCP test runners, IDE diagnostics and the PowerShell tool are not read.
• Edits outside the agent are invisible. A file you change in your editor has no transcript event; there is no cross-check with git status or file mtimes.
• Subagents are judged separately. A subagent call counts as an edit for every rule (by default); what it touched is not parsed, and its verdict is not reused as the parent's evidence.
• A transcript that starts mid-history only warns. When the first message's parentUuid names an entry missing from the file, the evidence is known to be incomplete, so blocks become warnings.
• The transcript format is internal to Claude Code and pinned to 2.1.281 (docs/transcript-format.md); on a changed format the hook fails open, and its accuracy is unverified until re-pinned.
• Tamper protection is partial. It undoes only the four downgrades above. Other keys from an edited config still apply, including max_transcript_mb, check_subagents and subagent_skip_types (read before the transcript is parsed) and a rule's claims and keywords; an invalid config fails open with a warning.
• Out of scope: generic "done" or "implemented" claims, claims about external state no command can show ("the email was sent"), and Windows.
• The Codex adapter is experimental and audit-only, pinned to one openai/codex source commit and built from hand-written fixtures.

Uninstall or disable

claude plugin uninstall proof-of-done@proof-of-done   # remove it
claude plugin disable proof-of-done                   # keep it installed, turn it off

Or set PROOF_OF_DONE=off for Claude Code's hook processes, or enabled: false in .proof-of-done.yaml.

Roadmap

• Reuse a subagent's already-checked verdict as evidence for its parent's claim.
• Evidence from sources other than Bash (MCP test runners, IDE diagnostics).
• Pin the Codex adapter against a verified rollout log and consider a Codex-side hook.

Data and licence

All fixtures, templates, held-out, development and adversarial sessions are synthetic, written for this repository and licensed Apache-2.0 with the code (see fixtures/README.md). The only vendored third-party code is a pure-Python copy of PyYAML 6.0.3 (MIT, licence kept in src/proof_of_done/_vendor/yaml/LICENSE), so the hook needs no install step (ADR 3).

Apache-2.0, see LICENSE. Copyright 2026 Andrii Boiko.