sober

How often did your coding agent actually verify what it claimed?

sober reads the session logs that Claude Code and Codex already keep on your machine, finds every sentence where the agent said tests pass / check passed / fixed / built / rendered / pushed, and checks whether a matching command really ran after its last code edit — and what that command's output said.

$ pip install sober-cli && sober audit

Verification rate: 88.1%   (119 verified / 135 decided claims)
Sessions: 93   Claims about this turn's work: 170

  verified         119   a matching check ran after the last code edit and its output agrees
  unverified        16   the agent edited code, then claimed success without running a check
  contradicted       0   the check it ran says otherwise
  could not check   35   evidence is off-transcript (subagents, background, piped exit codes, remote runs)

By agent / model:
  claude-code claude-opus-5-5     86%  (131 decided, 29 could not check)

Those are the author's own numbers, from 93 real sessions. A team that hand-labelled 357 "tests pass" claims from 150 Claude Code sessions found 80 with no run backing them; most were stale: tests passed, the agent edited again, then reported the old pass as current. Benchmarks say the same thing about ML tasks (57% of frontier-agent runs take a planted shortcut, and telling the agent not to barely helps). sober gives you your own number, then stops the pattern with two hooks.

• Deterministic. Regex and transcript structure only. No LLM, no API key, nothing leaves your machine.
• Precision first. Anything it cannot see is could not check, never unverified. The flagged set was tuned over six rounds of double-blind labelling until two independent raters agreed with every flag (details in gate/REPORT.md).
• Reads the output, not just the exit code. pytest -q | tail -3 returns tail's exit code; sober reads the last N passed / N failed summary instead, and reports a contradicted claim when the summary disagrees with the sentence.

Install

pip install sober-cli      # Python ≥ 3.9, standard library only
sober audit                # Claude Code (~/.claude/projects) + Codex (~/.codex/sessions)
sober audit --agent claude-code --json findings.jsonl   # every claim with context
sober share                # writes ~/.sober/sober-card.svg (+ .png on macOS / with rsvg-convert)

Make it stop happening: sober guard

sober guard install --dry-run   # shows the hooks it would add to ~/.claude/settings.json
sober guard install
sober guard status | uninstall

Two Claude Code hooks, both tiny and deterministic:

If your project already has a verify skill or script, the Stop hook recognises its run as evidence; it does not fight your existing workflow.

How a claim is judged

1. Scope: one turn. Everything the agent did between your prompt and its reply.
2. Last code edit. Edit/Write tools, or sed -i, cat > file, heredocs that write files. Edits to docs, logs, notes, memory, .env, temp dirs or .agent/ state do not count.
3. Matching check after it. Tests for tests pass; lint/type/build/check tools for check passed; render tools for rendered; git push / kaggle … push for pushed; tests, checks or renders for fixed (a generic script run is not evidence that a fix works).
4. Its output. The last summary line decides: 12 passed backs the claim, 2 failed contradicts it, fail 0 is a pass. A narrowed run (pytest -k, one file) cannot back "all tests pass".
5. Everything invisible → could not check: subagents, background tasks, your own terminal, piped commands with no summary in the output, remote runs (CI, Kaggle), syntax-only checks.
6. Not a claim: plans ("I'll run the tests"), questions, quotes, code blocks, sentences reporting a previous turn.

Forked sessions are de-duplicated so the same turn is counted once.

Precision

Every rule change was followed by a labelling round: the author and an independent Claude agent (fresh context, no access to the author's labels) each judged every flagged claim.

Recall is not measured. sober would rather say could not check 35 times than flag one verified claim.

What it is not

• Not a judge of correctness. A verified claim means the check ran after the edit and its output agreed with the sentence; it does not mean the code is right.
• Not a leaderboard of models. The per-model table is your sessions only.
• Not Anthropic's or OpenAI's. Independent, MIT, unaffiliated.

Related work

claude-guard-hooks and groundtruth block unbacked "done" claims with keyword matching; superpowers asks for verification in a skill (prompt-level). sober differs in reading command output, scoring your history across agents and models, and refusing to count what it cannot see.

中文说明见 README.zh-CN.md。