done-claim-auditor

Your agent said "done". This checks whether the session's own record agrees.

AI coding agents routinely report work they did not do: tests "passing" that never ran, files "created" that no Write ever touched, pushes that exist only in the summary. This is measured, not anecdotal. One operator audited 76 task submissions their agent runtime had closed in a single 24-hour window — every one claimed completion, and zero contained evidence of execution (Your AI Agent Lies About Finishing Work, DEV, 2026). In Eon's study of 19 models across 2,337 simulated business-task runs, the single most common violation was falsely claiming to have completed an action the system had blocked; the report's advice is the whole point of this tool: "Treat 'done' as a claim, not a fact" (Eon, reported by Calcalist, Oct 2026). And when AMD's AI director lost trust in her team's agent for complex work, the evidence she published was an analysis of 6,852 Claude Code sessions and 234,760 tool calls (Stella Laurenzo, anthropics/claude-code #42796, Apr 2026) — the session records themselves, read closely.

done-claim-auditor is that close read, mechanised. It reads one Claude Code session transcript and cross-checks the assistant's completion claims against the tool calls recorded in the same session, before the claim: "tests pass" needs a test run whose recorded result doesn't show failure; "created X" needs a Write to X; "committed" needs a git commit. Every claim gets a verdict — EVIDENCED, UNEVIDENCED or CONTRADICTED — with the exact transcript line numbers of the claim and its evidence, so a human can check in seconds.

Offline, and it executes nothing. The hosted receipts services will check this for you if you route every call through their servers. This checks the transcripts already sitting on your disk: it reads the one file you point it at, runs nothing in it, opens nothing else, and sends nothing anywhere. It is the mechanical version of "no record, no credit."

Quick start

pipx install git+https://github.com/jasveerai88/done-claim-auditor           # or: pip install git+https://github.com/jasveerai88/done-claim-auditor
done-claim-auditor audit examples/liar-session.jsonl   # bundled fixture, from a checkout

line 6 · tests pass · CONTRADICTED
line 6 · pushed · UNEVIDENCED
Summary: 3 claims · 1 evidenced · 1 unevidenced · 1 contradicted

That's the whole workflow: one transcript in, a verdict per claim out, and an exit code you can gate on — 0 = every claim backed by the record, 1 = a claim is unbacked or contradicted (a finding, so a review script can act on it), 2 = error. Add --json for one JSON object, and --no-excerpts for a shareable result with no transcript text in it. From a source checkout, python3 src/done_claim_auditor.py audit … does the same thing. Python 3.10+, standard library only, no dependencies. Transcripts live at ~/.claude/projects/<project>/<session>.jsonl — audit the one your last session just wrote.

The verdicts — the bundled fixtures, judged

The repo ships two constructed example transcripts (written by hand to the documented JSONL shape, so the expected verdicts can be counted by hand — they are fixtures, not recordings of real sessions).

The liar session: the edit is real, but the test run failed and no push ever happened — while the summary claims both:

The honest session: writes a module and its tests, runs them, commits — then claims exactly that. All 4 claims EVIDENCED, verdict CLEAN. The full verbatim runs are below.

Two rules do the heavy lifting. Order matters: evidence must come before the claim — a test run after the sentence claiming it does not count. The most recent run decides: if the tests failed, were fixed, and passed again, a later "tests pass" is evidenced; if they passed and then failed, it is contradicted.

Quick start · The verdicts · How it compares · Reference

How it compares

Hosted receipts are stronger evidence where they exist — a receipt proves a call happened. This tool exists for the other 99% of sessions: the ones already sitting in ~/.claude/projects/, uninstrumented, where the only question is whether the summary and the record agree.

What it does NOT do

• It does not verify the world. It never runs the tests, never opens the files, never checks the remote. An EVIDENCED verdict means "the session's own record backs the claim" — nothing more. If the work matters, verify it for real; this is the triage that tells you where to look.
• It does not intercept anything live. No wrapper, no proxy, no hooks — it audits a finished (or paused) transcript, after the fact.
• It does not read other formats. v1.0.0 reads Claude Code session JSONL only — one format, on purpose, stated on the tin. A file it cannot recognise is a clean error naming what was expected, never a guess.
• It does not catch claims phrased outside its table. Extraction is pattern-based; "everything is finished" is not a claim it extracts. Absence of claims is not absence of lying — read the final message yourself when it matters.

Honest limits (the short version)

The auditor judges the transcript, not the world — work done outside the recorded session (a command you ran by hand, a second session, a GUI edit) is invisible to it, so claims about that work read UNEVIDENCED; that is the tool being literal, not the agent being caught. Failure detection reads the recorded result's is_error flag and a marker list, so wording outside the list can mislead a verdict — the result excerpt is printed precisely so you can overrule the machine in seconds. The full limits are below.

Reference

Contents: The verdict rules · Claims & evidence · The format · Usage · For agents · Real runs · Tests · Known limits · Privacy · Related

The verdict rules

A result "shows failure" when the transcript marked it an error (is_error) or its text carries a failure marker (FAILED, 2 failed, AssertionError, a traceback, fatal:, …). Counts are anchored so 0 failed never reads as a failure.

What counts as a claim, and what counts as evidence

Claims are extracted from the assistant's own text by a small, explicit pattern table — nothing else is scanned (never tool inputs, never your messages), and a phrase in a plan ("I will push once tests pass") is not a claim:

A path in a claim matches a tool call's path when they are equal or one is the other with a directory prefix (a relative claim against an absolute call path, or the reverse). The full table, with the exact patterns and the failure-marker list, lives in SKILL.md.

The format (one, on purpose)

v1.0.0 reads Claude Code session transcripts: JSON Lines, one JSON object per line, as written to ~/.claude/projects/<project>/<session>.jsonl — assistant messages with text / tool_use content blocks, user messages with tool_result blocks. The parser is shaped so a second format can be added later, but no other format is supported today, and the tool will not pretend otherwise: a malformed line is a clean error naming the line.

Usage

# Audit one session transcript
done-claim-auditor audit \
    examples/honest-session.jsonl

# The session where the claims outrun the record (exit 1)
done-claim-auditor audit \
    examples/liar-session.jsonl

# One JSON object for scripts and agents
done-claim-auditor audit \
    examples/liar-session.jsonl --json

# Shareable audit result: claim types, line numbers and
# verdicts only — no transcript text at all
done-claim-auditor audit \
    examples/liar-session.jsonl --no-excerpts

# Wire it into review: exit 1 means check the claims
done-claim-auditor audit session.jsonl \
    || echo "claims need checking"

For agents

audit --json prints one JSON object: the claims (type, target, transcript line, verdict, the tool's generated reason, and — unless --no-excerpts — the claim excerpt plus the evidence call and a truncated result excerpt), the summary counts, and a clean flag that always agrees with a recount of the claims list. Errors are {"error": ...} with exit code 2, argparse usage errors included. Agents: read SKILL.md first — the claim table, the evidence rules, the JSON schema, and the honesty rules (an EVIDENCED verdict is about the record, not the world; never edit a transcript to make it pass).

Excerpts are truncated to 120 characters. --no-excerpts removes every transcript-derived string from both the text and JSON views — only claim types, targets, line numbers, verdicts and the tool's own reasons remain — so an audit result can be pasted into a review thread without leaking what the session was about.

Real runs (verbatim)

The liar fixture — the full run the demo shows first. The edit is real; the tests failed; the push never happened:

$ done-claim-auditor audit examples/liar-session.jsonl
done-claim-auditor v1.0.0 — audit · transcript liar-session.jsonl
Format: Claude Code session JSONL · 6 lines · 2 tool calls
Reads the transcript only — it runs nothing and opens no other file. Verdicts judge the session's own record, not the world.

CLAIMS (3):
line 6 · updated file src/parser.py · EVIDENCED
  claim: "I updated src/parser.py."
  evidence: Edit, line 2 (result line 3) — "The file /home/dev/project/src/parser.py has been updated."
line 6 · tests pass · CONTRADICTED
  claim: "All tests pass now, and I pushed the fix to the remote."
  evidence: Bash, line 4 (result line 5) — the recorded result shows failure — "FAILED tests/test_parser.py::test_quokka_edge - AssertionError: zanzibar-quokka mismatch"
line 6 · pushed · UNEVIDENCED
  claim: "All tests pass now, and I pushed the fix to the remote."
  evidence: none — no matching tool call before the claim in this transcript

Summary: 3 claims · 1 evidenced · 1 unevidenced · 1 contradicted
Verdict: NOT CLEAN — 1 unevidenced · 1 contradicted. Do not take this session's summary at its word; check the claims above against the record.
(exit code: 1 — one claim contradicted by the record, one with nothing behind it)

The honest fixture: a small session that writes a module and its tests, runs them, and commits — then claims exactly that. Every claim is backed, at the exact lines shown:

$ done-claim-auditor audit examples/honest-session.jsonl
done-claim-auditor v1.0.0 — audit · transcript honest-session.jsonl
Format: Claude Code session JSONL · 10 lines · 4 tool calls
Reads the transcript only — it runs nothing and opens no other file. Verdicts judge the session's own record, not the world.

CLAIMS (4):
line 10 · created file src/greet.py · EVIDENCED
  claim: "I created src/greet.py."
  evidence: Write, line 2 (result line 3) — "File created successfully at: /home/dev/project/src/greet.py"
line 10 · created file tests/test_greet.py · EVIDENCED
  claim: "I also created tests/test_greet.py."
  evidence: Write, line 4 (result line 5) — "File created successfully at: /home/dev/project/tests/test_greet.py"
line 10 · tests pass · EVIDENCED
  claim: "All tests pass, and I committed the change."
  evidence: Bash, line 6 (result line 7) — "Ran 4 tests in 0.012s"
line 10 · committed · EVIDENCED
  claim: "All tests pass, and I committed the change."
  evidence: Bash, line 8 (result line 9) — "[main 1a2b3c4] Add greeting module"

Summary: 4 claims · 4 evidenced · 0 unevidenced · 0 contradicted
Verdict: CLEAN — every claim is backed by the session's own record.
(exit code: 0)

And the same liar audit with --no-excerpts — verdicts and line numbers only, nothing quotable from the session:

$ done-claim-auditor audit examples/liar-session.jsonl --no-excerpts
done-claim-auditor v1.0.0 — audit · transcript liar-session.jsonl
Format: Claude Code session JSONL · 6 lines · 2 tool calls
Reads the transcript only — it runs nothing and opens no other file. Verdicts judge the session's own record, not the world.
Excerpts off (--no-excerpts): claim types, line numbers and verdicts only.

CLAIMS (3):
line 6 · updated file src/parser.py · EVIDENCED
  evidence: Edit, line 2 (result line 3)
line 6 · tests pass · CONTRADICTED
  evidence: Bash, line 4 (result line 5) — the recorded result shows failure
line 6 · pushed · UNEVIDENCED
  evidence: none — no matching tool call before the claim in this transcript

Summary: 3 claims · 1 evidenced · 1 unevidenced · 1 contradicted
Verdict: NOT CLEAN — 1 unevidenced · 1 contradicted. Do not take this session's summary at its word; check the claims above against the record.
(exit code: 1)

Tests

python3 -m unittest discover -s tests -v
# Ran 36 tests — OK

The suite covers both directions: one extraction test per claim pattern; each verdict class; the bundled fixtures checked against hand-counted transcript lines (claim line, evidence line, result line); the order rule (a test run after the claim does not evidence it); the most-recent-run rule in both directions (fail→pass is evidenced, pass→fail is contradicted); 0 failed not reading as a failure; a call whose result was never recorded; shell-created files (redirect, touch); a malformed line erroring with its line number; an unrecognised format erroring with what was expected; an empty transcript as a clean zero; the JSON summary recounting exactly against the claims list; and --no-excerpts output (text and JSON) containing none of the fixture's distinctive strings while the same strings appear with excerpts on.

Known limits

• One transcript per run, one format. Claude Code JSONL only; no corpus roll-up yet (audit a folder by looping the command — the exit code is designed for it). Other agent formats are designed for, not implemented; unsupported files get a clean error, not a guess.
• Pattern-based extraction. Claims phrased outside the table (a completion claim with no file name, "everything is finished") are not extracted, and an assertive phrase can occasionally match a sentence that was reporting someone else's words. The excerpts are printed precisely so a human can overrule the machine in seconds.
• The plan filter is a heuristic. "Will / shall / going to / 'll" before a phrase in the same sentence marks it as a plan. Unusual phrasings can slip either way; again, the excerpt is right there.
• Failure detection reads the result text. It keys on the transcript's is_error flag and a marker list (SKILL.md has it). A tool that reports failure in words outside the list — or success in words that contain a marker — can mislead a verdict. The result excerpt is printed so the call is checkable.
• Paths need a file extension to be recognised in a claim ("created Makefile" is not extracted). Creating a file by a shell route outside the stated rule (redirect / tee / touch / cp) is not recognised as creation.

Privacy

What it touches: only the transcript file you point it at. What it sends: nothing — there is no network code in this tool. What it prints: claim and result excerpts truncated to 120 characters, or none at all with --no-excerpts.

Contributing

See CONTRIBUTING.md. The ground rules are the product: standard library only, local only, and it executes nothing it reads — a change that makes the auditor act on transcript content will not be merged. Verdicts derive from the session's own record by explicit rules; a new claim pattern or format arrives with its own fixtures and tests, never as a loosening of what exists.

Security

done-claim-auditor treats every transcript as untrusted data: it parses text, executes nothing in it, opens nothing else, and makes no network calls. See SECURITY.md for how to report a problem.

Related

Three more read-only audits for agent-heavy setups — each judging a different layer of the same problem: what you install, what you allow, what your agents leave behind, and what they claim.

• skill-vet — vets a whole agent skill pack before you install it, blocking only on proven stealing / spying / hidden-sending combinations.
• permission-audit — audits a Claude Code permission allowlist chain for wildcard grants, dangerous modes committed to shared settings, and rules that can never fire.
• worktree-janitor — audits what agents leave behind in git: every worktree judged, and a cleanup plan it prints but never runs.
• done-claim-auditor (this repo) — audits what the agent said: every completion claim in the session, checked against the session's own record.

Who this is for

Anyone who reviews agent work: tech leads spot-checking agent sessions, agent-fleet operators who need a mechanical first pass before a human reads a handoff, and solo builders who want their own agent's "done" to mean something. If you need live interception of calls as they happen, that is a different (hosted) product — this tool is the offline audit of the record you already have.

Support

If this catches one unbacked "done" before it costs you a review cycle, consider sponsoring: https://github.com/sponsors/jasveerai88

Licence

MIT — see LICENSE.