buddie

Checks that what an AI agent tells you is true, before you read it. When Claude quotes a page, buddie looks the words up in a snapshot of that very page. When it states "merged", "CI is green" or "tests passed" with an anchor, buddie asks GitHub, git and the session log. No model takes part in the check, so the agent cannot talk its way past it, and every check ends in a receipt anyone can re-run.

In Claude Code the plugin checks each answer as it ends. A quote that is not on its page goes back to Claude with the closest real passage, before you see the answer:

buddie FAIL: quotes 1/2 found, 1 not found
- quote L3.1 not found in https://status.example.com/incidents/42: “elevated error rates on the API between 14:58 and 15:41 UTC” (closest 0.98: “elevated error rates on the API between 14:59 and 15:41 UTC”)

An answer with nothing to check passes untouched. Why it matters: in 929 checkable quotations from published deep-research reports, one in five was altered or absent from the page it cited (What we found).

Receipts for agent work. In a multi-agent setup an executor reports to an orchestrator, and the orchestrator reports to a human. Every hand-off restates claims such as "the source says X", "tests passed" or "PR merged", and a wrong claim travels on. buddie checks those claims at the hand-off against primary evidence: the bytes of the cited page, the repository, the GitHub API and the session transcript. No model takes part in the check, so the agent being checked cannot talk its way past it. Every check ends in a receipt that anyone can re-run.

· Python ≥ 3.11 · no dependencies · Apache-2.0 · Claude Code and Codex plugin, Claude Desktop extension, MCP server (stdio and HTTP) and CLI

This repository was called verbatim until October 2026. verbatim is now buddie's quote module (buddie.verbatim, command buddie-verbatim; until 0.8 it was a top-level verbatim package and command, which clashed with an unrelated verbatim on PyPI); old links redirect.

Why this matters for oversight

An agent's account of its own work is easy to accept and expensive to re-check. A quote that was altered, a test run that never happened or a PR described as merged while it is still open all pass as evidence when the reader cannot re-open every source. This is a narrow but measurable form of unfaithful reporting. buddie checks the account against records the agent did not write, without access to its reasoning. That makes it a small piece of scalable oversight that still works when the overseer is weaker than, or not trusted by, the agent.

What it checks

The executor writes the anchors. Lines that state work in prose without an anchor ("merged", "CI is green", "tests passed", "41 passed") are listed in the receipt (prose_work). In a project whose buddie.toml blocks prose, the hook returns them in a final answer with ready anchors (PRs named in the text or tool output, the run that printed the count), and a buddie: disclosure does not let them through (0.7.1). A negated claim ("not merged yet") is not one.

Quick start

pip install buddie
cd examples/quickstart
buddie-verbatim add https://status.example.com/incidents/42 page.html   # or let buddie fetch the page itself
buddie verify report.md --no-fetch

BLOCK quote L3.1 not found in https://status.example.com/incidents/42: “elevated error rates on the API between 14:58 and 15:41 UTC” (closest 0.98: “elevated error rates on the API between 14:59 and 15:41 UTC”)
BLOCK quote L7.1 not found in https://status.example.com/incidents/42: “the root cause was a misconfigured load balancer” (closest 0.50: “. The rest of the article discusses unrelated bac”)
buddie FAIL: quotes 1/3 found, 2 not found

A claim about work, checked against the session transcript (examples/work):

cd examples/work
buddie verify report.md --transcript session.jsonl --no-github

BLOCK anchor line 3 ⚓ run:pytest -q => 41 passed: output has no '41 passed'; it ends: ….......................................... 1 failed, 40 passed in 3.1s
buddie FAIL: effects 0/1 hold, 1 broken

The exit code is 1 on FAIL, so the check can gate a CI job, a hook or a hand-off. A wrong digit shows up as a close passage (similarity ≈ 0.98), an invented sentence as a distant one (≈ 0.5).

In Claude Code

/plugin marketplace add tsyyan/buddie
/plugin install buddie@buddie

The plugin brings four things:

• a hook (hooks/hooks.json) on PreToolUse for messages that leave the session (SendMessage) and on Stop; examples/hooks/projects.json adds matchers for other reply tools. On FAIL it returns the report to the model with the reasons, before anyone reads it. Since 0.6 it also counts live subagents (SubagentStart/SubagentStop); when none is left, the orchestrator is asked once for a summary built by code and a choice card for the next steps, and the person's pick (AskUserQuestion) goes to the journal, where the anchor choice:<id>=<step> is checked;
• an MCP server (stdio, no SDK): verify_report, snap, source_text, summary;
• skills: skills/buddie for executors and orchestrators, skills/verbatim for quoting from snapshots;
• commands you run yourself: /buddie:check [file or text] checks a text now (your last answer by default), /buddie:status shows what is checked in this project and what was returned lately, /buddie:init turns on the project's full checks (buddie.toml, starting in intro mode).

The hook runs through hooks/run.sh, which starts the first of python3, python and py -3 that is Python 3.11+, so it works on Linux, macOS and Windows (Claude Code runs hooks there in Git Bash).

Optional setting: a GitHub token for pr:/ci: anchors (the plugin asks for it and stores it as a secret; it is sent only to api.github.com). Without it GitHub allows 60 lookups an hour and private repos can't be checked.

Without Claude Code: python3 run.py verify REPORT.md --repo . from a clone works without installing anything.

Other clients

Releases and catalog listings: docs/release.md, docs/catalogs.md.

Levels and the intro mode

Right after installing (no buddie.toml in the repo), the hook returns an answer only for what is false: a quote that is not on the page it links to, or an anchor that does not hold. An answer with no quotes and no anchors passes untouched, and so does one whose source could not be read. Numbers of cited sentences that are not on their page only warn. Asking for anchors on every final answer, prose about work and the context limit are a project's choice, made with buddie init.

buddie init writes buddie.toml in the repo with the default level of each class of claims and turns the intro mode on. Each class is block (the text goes back to the model), warn (the text leaves; you see what the gate would have said) or off:

[gates]
prose = "block"

[intro]
finals = 20

In the intro mode nothing is returned: each would-be return is a warning that says what it would have returned. It ends by itself after finals final answers with no false return reported; buddie intro --false reports one and restarts the count, buddie intro shows where it stands, buddie enforce ends it now. Without buddie.toml the "Without" column applies and there is no intro: nothing that is merely unanchored is returned, so there is nothing to get used to.

The loop

The hook also learns from its own returns; none of it is written by the model:

• How the gate was passed. A return logs reasons; the next attempt of the same tool in the session logs gate: per flagged line proved, removed or kept, and the way through (proof, removal, mixed, disclosure, returned).
• Lessons at start. On SessionStart the hook prints a summary of past sessions' journal: how many messages came back, for what, and the shape of a line that passes (at most ten lines).
• Ready anchors. A return offers pr:, ci: and run: anchors that already hold (checked now), to copy onto the line instead of rewording it.
• Measure. buddie journal measure --sessions 10: the share of sessions whose first checked message passed, and how returns were passed (proof vs removal).
• Context gate. Past BUDDIE_CONTEXT_LIMIT tokens of context a non-final reply is returned with a reminder to hand the result over; final answers, hand-offs to another session and Stop are never held.

Receipt v0

Each check produces an in-toto Statement: the subject is the SHA-256 of the report text; the predicate (predicateType: https://github.com/tsyyan/buddie#receipt-v0) lists the verdict of every quote with the snapshot SHA-256 and offset, the status of every anchor, and the outcome:

ci: and run: receipts carry a note: an anchor shows what happened (a PR in that state, checks on that commit, a command and its output), not that the code is correct. A green CI on weak tests stays weak.

What we found

Deep-research reports quote wrongly. On 929 checkable quotations from published deep-research reports, labelled blind, 212 (22.8 %, 95 % CI 20–26 %) were altered by the agent or absent from the cited source. verbatim flagged 207 of those 212.

Source: results/E013-stage2-summary/recount.json. "Checked" means the cited page could be read and the quote is long enough to search (≥ 12 characters).

Numbers too. In 100 random sentences of DeepResearch Bench I reports (Gemini, Grok, OpenAI, Perplexity), 28 of 103 checkable cited numbers (27.2 %) were not in their source (E009; two blind labellers, κ 0.956).

Where the quote check is weak. Measured out of sample (each version on reports it was not tuned on), false NOT_FOUND is 10 of 440 (2.3 %, CI 1.2–4.1 %). On a new genre, Search Arena answers of Perplexity, GPT-4o search and Gemini grounding with the rules unchanged, it is 29 of 423 (6.9 %) (E014). Only 6 of the 29 (1.4 %) are real quotations the search missed; the rest are words the agent itself put in quotation marks (role-play lines, satire, example prompts). Telling "is this a quote" from the report text alone does not transfer across genres; our ≤ 2 % target is not demonstrated for arbitrary answers. The same set had 69 agent errors (16.3 %).

On our own agents. We use buddie in a Claude-based multi-agent project (Claude Code threads, an orchestrator, GitHub Actions). There it found 4 automatic task launches that the orchestrator announced before the parent change had merged, and the launch rule was changed to start only from the post-merge record. All 8 claims about PRs and CI in earlier thread reports held, and 3 of 3 deliberately false anchors were caught in a live run. Before buddie, a two-hour audit of the same threads found 14 reporting errors, and none was fixed by reasoning without going back to the source. These runs use the project's private chat, so their raw data is not published; the numbers are reported as observed.

The labellers were Claude subagents working blind to the verdicts, not humans. Verdicts were committed before labelling and rules frozen by commit before each out-of-sample run. The protocol is in docs/verbatim-method.md.

The quote module: verbatim

verbatim saves the bytes of every cited source under their SHA-256, extracts the text a reader would see, and searches each quotation in it. Each quote is checked against its own link, the citation next to it in the same sentence, not just any link in the paragraph.

Verdicts: FOUND / FOUND_NORMALIZED, HIDDEN_ONLY (only in markup the reader does not see), FOUND_IN_COPY, UNCERTAIN, NOT_A_QUOTE, NOT_FOUND, SOURCE_UNAVAILABLE / NO_SOURCE; see docs/verbatim-verdicts.md. --access default tries lawful routes for unreadable pages in order: a headless browser (pip install 'buddie[browser]'), the Wayback Machine and Common Crawl, open-access copies by DOI, known syndicated copies. verbatim never bypasses paywalls, captchas or bot protection.

Configuration

What leaves your machine

• Cited pages. To check a quote the hook downloads the page it cites (BUDDIE_FETCH=0: stored snapshots only). Snapshots go to VERBATIM_STORE. Only --access default in the CLI or MCP server also asks archive.org, Common Crawl and open-access copies by DOI.
• GitHub API. pr: and ci: anchors call api.github.com, with BUDDIE_GITHUB_TOKEN or GITHUB_TOKEN if set (BUDDIE_GITHUB=0: no calls).
• Nothing else: no telemetry, no model calls. The full list of what is read, sent and kept is in PRIVACY.md.

The transcript and the journal

The transcript. The hook reads the current session's transcript (transcript_path, and its subagents' files next to it) because a claim about a command can only be checked against what the session actually ran: ⚓ run:pytest -q => 41 passed holds only if the transcript shows that call with that output. It also takes the token count for the context gate, snapshot stores the session named, and the commands behind ready anchors from there; the final answer at Stop comes from the event itself (last_assistant_message), and from the transcript only when the event has none. It reads the file in place; nothing from it is copied into the journal or sent anywhere. No other session, no memory, no chat history is read. To turn it off, set the plugin's "Read the session transcript" option to off, BUDDIE_TRANSCRIPT=0, or [transcript] read = false in buddie.toml: answers are still checked, and run: anchors are reported as unchecked.

The journal. Each check appends one line to BUDDIE_LOG (~/.cache/buddie/hook.jsonl): time, session id, tool, outcome, counts of verdicts, exit code and the SHA-256 of the checked text. It never holds the text. Promise lines («launch №N after PR #M») keep the row, the condition and the sentence's SHA-256; choice lines (the person's pick on a summary card or an ask_decision card) keep hashes of the picked steps and the numbers of a typed answer, and a decision card line keeps its options' label hashes and queue numbers, which is enough for choice:<id>=<step> to be checked. [journal] text = true in buddie.toml (or BUDDIE_JOURNAL_TEXT=1) also keeps those sentences and answers. BUDDIE_LOG=0 turns the journal off; lessons at SessionStart then have nothing to sum.

Scope and limits

• It checks that words exist in the source, not that they support the claim around them. A real quote attached to the wrong date or event still passes.
• Unquoted paraphrase is not checked. Numbers are checked only in a sentence with a link, and a missing one is a gap by default (out of sample 6 of 139 checkable numbers were false NOT_FOUND).
• Pages change: a verdict holds for the snapshot hash and time it names. pr: and ci: judge the state now; write merged@SHA and ci: on a SHA for history.
• run: trusts the transcript, which the harness writes on the same machine as the agent. Its format (Claude Code jsonl) is undocumented; other agents are not supported. A command that prints a prepared file passes: the anchor says "the command answered this", not "the test passed".
• Work-claim and final-answer rules match Russian and English phrasing. By default the final answer is the last message at Stop; set BUDDIE_FINAL or final_mark to mark it in replies sent earlier.
• The gates cost turns: in the E016 pilot (12 tasks with traps per configuration) a session took 6.1 turns on average without buddie and 15.1 and 12.6 with the 0.4 and 0.5 gates. That pilot did not measure the main effect: there were no false "done" claims in any configuration, traps were too easy. Whether buddie makes agents misstate less is not yet measured.

Reproduce

pip install -e . pytest && pytest      # 314 self-contained tests; 44 more need third-party snapshots or the lab repo and skip
python results/E008-drb2-fresh/compare.py   # scripts are kept as they ran; paths assume the lab layout, see results/README.md

Development happens in this repository. Until 0.8.2 it was exported by script from a private lab repository, where the experiments in results/ were run; results/PROVENANCE.json is the record of that last export (lab@ed8d971): every file with its SHA-256 and lab path. We do not redistribute page snapshots, because they are other people's pages; verdict files keep each page's URL and SHA-256, so a re-fetch shows whether the page changed. Report text from DeepResearch Bench I/II is included under its Apache-2.0 license. Sets whose answer text we may not redistribute (ydc-deep-research-evals: no license; Search Arena: answers under the providers' terms) are published as ids, labels, verdicts and hashes only. See NOTICE and results/README.md.

Plugin evals

evals/ runs claude plugin eval with the plugin and without it (2 runs per arm, Claude Opus 5.5, 2026-10-20). Each case is an orchestrator passing a subagent's report on to the person:

With the plugin the person got the truth in every trap run; without it, in none. The cost is turns: 3.7 to 5 per run against 1, about twice the price. These are four hand-made cases, two runs each, where the report is told to be passed on as written: they show what the gate does when a false claim is there, not how often agents make one.

Status

buddie 1.1.1 with verbatim 0.3.8. 1.0 fixes the interfaces: the receipt (predicateType …#receipt-v0), the CLI, the MCP tools and buddie.toml; changes to them follow semantic versioning. It checks claims about quotes, numbers, PRs, CI and runs against primary data. Its effect on agent behaviour is still being measured: a gate that returns a false report does not by itself show that agents misstate less. Next: current Claude models with web search and as coding sub-agents, measured on how often their reports misstate sources, tests or merges; and an intervention study, where the agent runs buddie before it reports and we check whether false claims drop or the agent shifts to unverifiable wording.

Cite

See CITATION.cff.