honestly

Coding agents report "tests pass" without always running them. honestly checks whether a matching test or build actually ran after the last code edit, and blocks the claim when it did not.

Claude Code and Codex edit code, then report tests pass, fixed, build ok, pushed. Some of those reports were never run: the agent remembers an earlier pass, edits again, and repeats the old result as current. The reply looks the same either way, so the error surfaces later, in CI, on a colleague's machine or in production, where it costs hours instead of seconds. honestly moves the check back to the cheapest moment: before the sentence is sent.

Install the guard

pip install honestly          # Python 3.9+, standard library only, nothing leaves your machine
honestly guard install        # adds two hooks to ~/.claude/settings.json (--dry-run shows them first)

From then on, when the agent is about to end a turn with an unverified claim, this appears in the chat, in the language of the conversation:

honestly stepped in: Claude Code (claude-opus-5-5) in project docpilot, answering "fix the login bug"
said "Fixed the login bug, all 12 tests pass" but ran no check after editing login.py.
It has been told to run one first.

The agent then runs the check and starts its next reply by owning up: "honestly caught me: I said tests pass before running them; I have now run them and the result is 12 passed." If the check fails, it sees the failure and fixes the code. If no check can be run, it must rewrite the sentence as unverified. The second hook refuses a test or build command whose result is cut off by | tail or | head, since a failure would look like a pass.

honestly catches              # what guard stopped in the last 7 days, by project and model
honestly guard status         # or: uninstall (removes exactly the entries it added; a backup is kept)
honestly guard notify on      # optional: also show each catch as a desktop notification (off to stop)

If your project already has a verify skill or script, the Stop hook recognises its run as evidence; it does not fight your existing workflow.

Audit your history

honestly audit                # reads ~/.claude/projects (Claude Code) and ~/.codex/sessions (Codex)
honestly audit --since 30d    # last 30 days only (also 8w, 3m)
honestly explain 3            # the full timeline behind the 3rd flagged claim: edit, tools after it, why
honestly share                # writes a card to ~/.honestly/ (SVG, plus PNG when Chrome, Edge or rsvg-convert is present)

Verification rate: 95.2%   (278 verified / 292 decided claims)
Sessions: 101   Claims about this turn's work: 362

  verified         278   the check or action behind the claim is on the transcript and succeeded
  unverified        14   it claimed success (tests pass, fixed, sent, deployed, saved...) with nothing behind it
  contradicted       0   the check or action it ran says otherwise
  could not check   63   evidence is off-transcript (subagents, background, piped exit codes, remote runs)
  weak               7   a visual fix judged by looking at a rendered image; not counted either way

Those are the author's own numbers from 101 sessions, with every flag confirmed by a second, independent rater. A team that hand-labelled 357 "tests pass" claims from 150 Claude Code sessions found 80 with no run behind them. Benchmarks report the same on ML tasks (57% of frontier-agent runs take a planted shortcut, and telling the agent not to barely helps). Light users will see few claims at first; the guard works from the first session regardless.

• Deterministic. Regex and transcript structure only. No LLM, no API key, nothing leaves your machine.
• Precision first. Anything it cannot see is could not check, never unverified. The flagged set was tuned over six rounds of double-blind labelling until two independent raters agreed with every flag (gate/REPORT.md).
• Reads the output, not just the exit code. pytest -q | tail -3 returns tail's exit code; honestly reads the last N passed / N failed summary and reports contradicted when it disagrees with the sentence.

Beyond code: sent, deployed, downloaded, saved, checked, scheduled

Agents also say I've emailed the maintainer, deployed to production, all 10 files downloaded, saved to report.csv, I checked the docs, scheduled the job. These have no code edit to anchor on, so the evidence window is the whole turn: did a tool call or command that performs the action run before the sentence, and did it succeed?

Precision rules learned from labelling 41 flags with a second independent rater (35 of them were summaries, background jobs or earlier turns): a turn with no tool calls is a summary and is never flagged (no edit this turn); the previous turn's tools count, since replies often open by reporting what the last turn did; a background task, subagent, browser action or another session makes the claim could not check; "checked" is never contradicted by one failed lookup. On the author's logs the seven new categories now yield 134 verified, 46 could not check and 0 flags, so what they catch on other people's logs is still an open question.

How a claim is judged

1. Scope: one turn. Everything the agent did between your prompt and its reply.
2. Last code edit. Edit/Write tools, or sed -i, cat > file, heredocs that write files. Edits to docs, logs, notes, memory, .env, temp dirs or .agent/ state do not count.
3. Matching check after it. Tests for tests pass; lint/type/build/check tools for check passed; render tools for rendered; git push / kaggle … push for pushed; tests, checks or renders for fixed (a generic script run is not evidence that a fix works).
4. Its output. The last summary line decides: 12 passed backs the claim, 2 failed contradicts it, fail 0 is a pass. Understands pytest, unittest, Jest/Vitest/Mocha, Go, Cargo, Maven/Gradle, .NET, tsc, ESLint, Ruff, Clippy, go vet and more. A narrowed run (pytest -k, one file) backs a claim only when the claim names the same count ("13 tests pass" ↔ 13 passed).
5. Visual fixes. "Fixed the overlap" followed by a render and a look at the screenshot is weak: shown separately, counted neither as verified nor unverified.
6. Everything invisible → could not check: subagents, background tasks, your own terminal, piped commands with no summary in the output, remote runs (CI, Kaggle), syntax-only checks.
7. Not a claim: plans ("I'll run the tests"), questions, quotes, code blocks, sentences reporting a previous turn.

Forked sessions are de-duplicated so the same turn is counted once.

Precision

Every rule change was followed by a labelling round: the author and an independent Claude agent (fresh context, no access to the author's labels) each judged every flagged claim.

Rounds 3–5 were intermediate fixes (heredoc parsing, variable paths, remote-run claims) and are listed in the report. The independent rater is a Claude agent in a fresh context with no access to the author's labels; it is still a model, not a human. A human second rater is the obvious next step, and the labelling sheets are available on request.

Recall. Two raters (author + independent agent) then labelled 66 claims the audit had not flagged. They found 2 and 3 misses respectively, 5 distinct: an experiment runner mistaken for a test, a "background" sentence mistaken for a remote run, a commit claim with no commit after the last edit, a test claim backed only by a background job, and a progress list whose check predated the edit. Four of the five are fixed in 0.2.0. Before the fixes the recall estimate on that sample was roughly 55–75% depending on the rater; the remaining known miss is the progress-list case. honestly still prefers could not check over a wrong flag, and the full labelling material is described in gate/REPORT.md.

What it is not

• Not a judge of correctness. A verified claim means the check ran after the edit and its output agreed with the sentence; it does not mean the code is right.
• Not a leaderboard of models. The per-model table is your sessions only.
• Not Anthropic's or OpenAI's. Independent, MIT, unaffiliated.
• The guard is Claude Code only: Codex has no equivalent stop-before-reply hook, so Codex sessions can be audited but not intercepted.
• Not a judge of test adequacy. It checks that a matching check ran after the edit and what its output said, not whether that check covers the lines that changed.
• Subagent and background runs are reported as could not check; they are not read yet.
• Tested on Linux, macOS and Windows (CI), Python 3.9–3.13. Rules were tuned on one developer's sessions (Python, JS, video tooling, Kaggle) and then against two external testers' logs, which is where the first false flags came from (an English bibliography title containing "fixed bed reactor"). Other stacks are covered by synthetic tests only; expect to report a false flag or two, and they are fixed the same day. Previously published as sober-cli.

Related work

claude-guard-hooks and groundtruth block unbacked "done" claims with keyword matching; superpowers asks for verification in a skill (prompt-level). honestly differs in reading command output, scoring your history across agents and models, and refusing to count what it cannot see.

中文说明见 README.zh-CN.md。