sober

How often did your coding agent actually verify what it claimed?

sober reads the session logs that Claude Code and Codex already keep on your machine, finds every sentence where the agent said tests pass / check passed / fixed / built / rendered / pushed, and checks whether a matching command really ran after its last code edit — and what that command's output said.

$ pip install sober-cli && sober audit

Verification rate: 91.5%   (140 verified / 153 decided claims)
Sessions: 96   Claims about this turn's work: 176

  verified         140   a matching check ran after the last code edit and its output agrees
  unverified        13   the agent edited code, then claimed success without running a check
  contradicted       0   the check it ran says otherwise
  could not check   16   evidence is off-transcript (subagents, background, piped exit codes, remote runs)
  weak               7   a visual fix judged by looking at a rendered image; not counted either way

By agent / model:
  claude-code claude-opus-5-5     86%  (131 decided, 29 could not check)

Those are the author's own numbers, from 96 real sessions. A team that hand-labelled 357 "tests pass" claims from 150 Claude Code sessions found 80 with no run backing them; most were stale: tests passed, the agent edited again, then reported the old pass as current. Benchmarks say the same thing about ML tasks (57% of frontier-agent runs take a planted shortcut, and telling the agent not to barely helps). sober gives you your own number, then stops the pattern with two hooks.

• Deterministic. Regex and transcript structure only. No LLM, no API key, nothing leaves your machine.
• Precision first. Anything it cannot see is could not check, never unverified. The flagged set was tuned over six rounds of double-blind labelling until two independent raters agreed with every flag (details in gate/REPORT.md).
• Reads the output, not just the exit code. pytest -q | tail -3 returns tail's exit code; sober reads the last N passed / N failed summary instead, and reports a contradicted claim when the summary disagrees with the sentence.

Install

pip install sober-cli      # Python ≥ 3.9, standard library only
sober audit                # Claude Code (~/.claude/projects) + Codex (~/.codex/sessions)
sober audit --since 30d    # last 30 days only (also 8w, 3m)
sober audit --agent claude-code --json findings.jsonl   # every claim with context; per-project breakdown is in the report
sober share                # writes ~/.sober/sober-card.svg (+ .png via rsvg-convert or a local Chrome)

Make it stop happening: sober guard

sober guard install --dry-run   # shows the hooks it would add to ~/.claude/settings.json
sober guard install
sober guard status | uninstall

Two Claude Code hooks, both tiny and deterministic:

If your project already has a verify skill or script, the Stop hook recognises its run as evidence; it does not fight your existing workflow.

How a claim is judged

1. Scope: one turn. Everything the agent did between your prompt and its reply.
2. Last code edit. Edit/Write tools, or sed -i, cat > file, heredocs that write files. Edits to docs, logs, notes, memory, .env, temp dirs or .agent/ state do not count.
3. Matching check after it. Tests for tests pass; lint/type/build/check tools for check passed; render tools for rendered; git push / kaggle … push for pushed; tests, checks or renders for fixed (a generic script run is not evidence that a fix works).
4. Its output. The last summary line decides: 12 passed backs the claim, 2 failed contradicts it, fail 0 is a pass. Understands pytest, unittest, Jest/Vitest/Mocha, Go, Cargo, Maven/Gradle, .NET, tsc, ESLint, Ruff, Clippy, go vet and more. A narrowed run (pytest -k, one file) backs a claim only when the claim names the same count ("13 tests pass" ↔ 13 passed).
5. Visual fixes. "Fixed the overlap" followed by a render and a look at the screenshot is weak: shown separately, counted neither as verified nor unverified.
6. Everything invisible → could not check: subagents, background tasks, your own terminal, piped commands with no summary in the output, remote runs (CI, Kaggle), syntax-only checks.
7. Not a claim: plans ("I'll run the tests"), questions, quotes, code blocks, sentences reporting a previous turn.

Forked sessions are de-duplicated so the same turn is counted once.

Precision

Every rule change was followed by a labelling round: the author and an independent Claude agent (fresh context, no access to the author's labels) each judged every flagged claim.

Recall. Two raters (author + independent agent) then labelled 66 claims the audit had not flagged. They found 2 and 3 misses respectively, 5 distinct: an experiment runner mistaken for a test, a "background" sentence mistaken for a remote run, a commit claim with no commit after the last edit, a test claim backed only by a background job, and a progress list whose check predated the edit. Four of the five are fixed in 0.2.0. Before the fixes the recall estimate on that sample was roughly 55–75% depending on the rater; the remaining known miss is the progress-list case. sober still prefers could not check over a wrong flag, and the full labelling material is described in gate/REPORT.md.

What it is not

• Not a judge of correctness. A verified claim means the check ran after the edit and its output agreed with the sentence; it does not mean the code is right.
• Not a leaderboard of models. The per-model table is your sessions only.
• Not Anthropic's or OpenAI's. Independent, MIT, unaffiliated.
• Tested on Linux, macOS and Windows (CI), Python 3.9–3.13. Rules were tuned on one developer's sessions (Python, JS, video tooling, Kaggle); other stacks are covered by synthetic tests only, so expect to report a false flag or two. Issues welcome.

Related work

claude-guard-hooks and groundtruth block unbacked "done" claims with keyword matching; superpowers asks for verification in a skill (prompt-level). sober differs in reading command output, scoring your history across agents and models, and refusing to count what it cannot see.

中文说明见 README.zh-CN.md。