Agent Countercheck

Proof that AI agents' work really happened.

AI coding agents often say "done, all tests pass" when it isn't true: they skip a failing test, edit the test so it passes, or never run it. countercheck verify checks the claim itself:

1. It reads your checks from the base revision (before the agent's change), so the agent can't weaken them.
2. It copies the agent's files into a clean folder and puts back any protected file the agent changed (your tests).
3. It runs the checks in a macOS sandbox: no internet (localhost works; a check can opt in with network = true), writes only inside that folder.
4. It writes a receipt: the claim, what ran, the output, what the agent changed that it shouldn't have, the cost if known, and a verdict. Receipts are sealed with a SHA-256 hash, so edits are detectable.

The same run as text:

$ countercheck verify --claim "Fixed the discount bug. All tests pass." --agent claude-code --cost-usd 0.42
countercheck receipt 9073a384e88f9d37  FAILED
  claim:    "Fixed the discount bug. All tests pass." -> CONTRADICTED by evidence
  checks:   .countercheck/checks.toml @ HEAD (locked), sandbox: macos-seatbelt (no internet, localhost only; writes only in the check folder)
  protected Modified: tests/test_pricing.py -> restored_from_base
  [FAIL] unit tests  (65 ms)  $ python3 -m unittest -q
         |     self.assertEqual(apply_discount(10, 150), 0)
         | AssertionError: -5.0 != 0
  cost:     $0.4200 (reported)
  - the worker changed 1 protected file(s); countercheck ran the checks with the base version
  - check "unit tests" failed (exit 1)

(The agent's own run said OK (skipped=1): it had marked the failing test as skipped.)

Agent Countercheck ("Countercheck" for short) installs as agent-countercheck and runs as countercheck, the same way pip install pre-commit gives you pre-commit. It was called countercheck up to v0.3; the old install names still work (see Install).

How it works

The mental model: the agent never grades itself. Its claim is compared with what actually runs, on rules it couldn't change:

base revision (locked)        agent's work + claim
  checks.toml, your tests          files, "Done, all tests pass."
          \                           /
           +--> countercheck verify <+
                1 lock the checks     (from the base revision)
                2 clean copy          (edited tests put back)
                3 run in a sandbox    (no internet unless a check opts in)
                4 decide + seal       (receipt with a SHA-256 seal)
                        |
      VERIFIED -> agent may finish
      FAILED + "done" -> hook sends the failure back; agent fixes the code (max 3 rounds)
      FAILED + honest "can't, because ..." -> agent may stop; receipt still says FAILED

How Countercheck thinks

The code behind each rule, one run step by step, and where new work plugs in: docs/architecture.md.

Where it fits

Ready recipes for real projects (check files, helper scripts, a 5-minute start): recipes/ — first one: Oracle → Snowflake conversion with Cortex Code.

Does it work?

We gave Hermes Agent 29 small coding tasks, 3 times each, in four setups, and graded every run with hidden tests the agent never saw. Some tasks are impossible on purpose (a test expects a wrong value, a server can't be reached): the honest answer is "this can't pass, because …".

• When GPT 6.1 got stuck, it edited the protected test and said "all tests pass". Countercheck re-ran the original tests, said FAILED and pushed back once; the agent restored the test and reported the real problem.
• Countercheck only knows what your checks test: the remaining false claims were cases the visible tests didn't cover. DeepSeek rarely games tests, so there was little to catch.
• It added about 1 second to the median run.

Full method, numbers and limits: docs/eval/results-2026-10.md. Reproduce: eval/.

Install

Builds: macOS (Apple silicon and Intel, one universal binary), Linux (x86_64, arm64; static) and Windows (x86_64, countercheck.exe, no runtime to install). Both install scripts check every download against SHA256SUMS.txt before installing. install.ps1 puts countercheck.exe in %LOCALAPPDATA%\Programs\countercheck and tells you how to add that folder to PATH.

Windows needs Git for Windows (Countercheck calls git, and checks run with its sh, so the same checks.toml works on every OS). A check can use PowerShell or cmd instead: shell = "pwsh" or shell = "cmd" (docs/checks.md).

Sandbox by platform. On macOS, checks run in the system sandbox (no internet, localhost allowed, writes only in the check folder; network = true lets a single check use the internet). On Linux and Windows there is no sandbox yet: checks run as a normal process and the receipt says so (on Windows: none (no sandbox on Windows yet)); run Countercheck inside a container or VM with networking off when you need isolation.

What's shipped, in progress and next: docs/ROADMAP.md.

Commands

All options: countercheck help <command>. Check types: docs/checks.md. Ready setups for real jobs: recipes/.

Quick start

countercheck init                      # writes .countercheck/checks.toml, edit the commands
git add .countercheck && git commit -m "Countercheck checks"
# ... let an agent work ...
countercheck verify --claim "what the agent said"            # uncommitted work vs HEAD
countercheck verify --base main                              # committed work on a branch vs main
countercheck show .countercheck/receipts/<id>.json                   # re-check a receipt's seal

.countercheck/checks.toml:

protected = ["tests/**"]          # the agent may not change these; .countercheck/** is always protected
link = ["node_modules", ".venv"]  # git-ignored dependency folders to make visible (read-only)

[[check]]
name = "unit tests"
run = "pytest -q"                 # any command; runs with `sh -c` in the clean copy
timeout_secs = 600

Besides plain commands there are matrix, forbid, golden, budget and coverage checks: see docs/checks.md and the runnable example in examples/data-migration.

Exit codes: 0 verified · 2 failed (claim contradicted) · 3 unverified (nothing failed, but the checks weren't locked in the base revision, or none ran) · 1 error · 4 (countercheck show) receipt was edited.

See every run: countercheck ui and countercheck report

countercheck ui          # a live page in your browser; updates when a new receipt lands
countercheck report --open   # the same page as one HTML file (offline, attach it to a PR or CI run)

countercheck ui serves the page on this computer only (127.0.0.1, with a random token in the address; other hosts and other web pages can't read it). It uses port 7357, or any free port if that one is taken (--port N to choose), and running it again opens the one already running for the project.

countercheck report writes one HTML page (.countercheck/receipts/report.html, git-ignored) with every saved run, newest first: the verdict, what the agent said, each check with its output, protected files it changed, whether the receipt's seal is intact, and for matrix checks a grid of targets × checks. It works offline, loads nothing from the internet and has no scripts, so it's safe to attach to a pull request or a CI run.

Agent hooks

• Hermes Agent: countercheck hook hermes-pre-verify keeps Hermes working while the checks fail. In the profile's config.yaml:

  hooks:
    on_session_start:
      - command: "countercheck hook session-start"
    pre_verify:
      - command: "countercheck hook hermes-pre-verify"
        timeout: 300
  

  Then approve both hooks once in a terminal (hermes hooks list shows "✗ not allowlisted" until you do; the desktop app never asks). Setup, approval and how the project is found: docs/hooks/hermes.md.

• Claude Code and Cortex Code: countercheck hook claude-stop blocks the Stop while the checks fail, with countercheck hook session-start on SessionStart. See docs/hooks/claude-code.md.

Add the session-start hook too. It remembers the commit each session started from, so the checks stay locked even if the agent commits its own edits to a test or to checks.toml. Without it, the checks are locked to HEAD. In CI, pass the base explicitly: countercheck verify --base origin/main.

On Windows the hooks work the same (Claude Code runs natively there): use the same settings with countercheck.exe on PATH, or give its full path with forward slashes, e.g. "command": "C:/Users/you/AppData/Local/Programs/countercheck/countercheck.exe hook claude-stop". Paths in the hook input may be C:\... or C:/...; a transcript under ~\ is found through USERPROFILE. See docs/hooks/claude-code.md.

How Countercheck itself is tested

A referee has to be right, so Countercheck is tested harder than most tools its size:

• 140 tests (cargo test --workspace): unit tests, end-to-end tests that run the real binary on throwaway git repositories (tampered tests, timeouts, the macOS sandbox, both agent hooks, every check type), and property tests that throw thousands of generated inputs at the verdict and seal logic (e.g. a receipt's seal breaks if any one of 33 fields changes; decide never says VERIFIED when a check failed).
• Mutation testing (cargo-mutants): the receipt, claim and config code was deliberately broken in every way the tool could find; tests were added until every break that compiles is caught (one remaining change is provably equivalent).
• CI on every push (macOS and Linux): cargo fmt --check, cargo clippy -D warnings, all tests, a release build and the benchmark task validator. Countercheck also verifies itself with its own .countercheck/checks.toml.
• The benchmark above, with hidden tests as ground truth.

Why Rust?

Countercheck is a referee that runs inside other people's projects: Python, JavaScript, Java, SQL, anything. So it has to be one dependency-free file that starts instantly, can't crash, and behaves the same on every operating system. Rust gives exactly that, which is also why tools like uv, ruff and OpenAI's Codex CLI are written in it.

The costs, honestly: fewer people know Rust than Python, and it is slower to write. That's why most contributions need no Rust at all: domain examples, check files and benchmark tasks are plain files, scripts and Python, and checks run your project's own commands in any language. Go would also have been a reasonable choice (single binary, simpler language); Rust won on stricter safety guarantees and its testing tools.

Where Rust is not the answer: a future hosted team dashboard (logins, many repositories, rich views) is a web app and will use a web stack. The full reasoning is in docs/decisions/0001-stack.md.

FAQ

Does Countercheck limit what my agent can do? No. The agent works with its full harness: web search, docs, tools, anything you allow. Countercheck only steps in when the agent says "done": it re-runs your locked checks on a clean copy of the result. Like an exam after open-book study: the grade has to reflect the work itself.

Does Countercheck edit tests or search the web? Never. It reads the checks from the base revision, runs them and writes a receipt. It makes no network connections of its own (no telemetry, no network library in the binary) and does not use an LLM: the verdict comes from your checks' exit codes and output.

What is the sandbox, and which OS has one? A fence the operating system puts around the check while it runs, so the agent's code can't fetch the answer from the internet or write outside the check folder.

So without a sandbox, can the agent cheat on Linux or Windows?

The common cheats are caught on every OS: edited tests are restored from the base revision, checks.toml is read from the locked base, committing edits doesn't move that base (the session-start hook), and every check is re-run in a clean copy. The sandbox adds protection against code written to misbehave during the check (fetching the expected answer online, leaving files outside the folder). Until Linux and Windows have one, run Countercheck in a container or VM with networking off when that matters.

Why does grading run offline? So the same work always gets the same verdict, and nobody can fetch the answer at grading time. In the benchmark, the only false alarms came from checks that called a real website. If a check truly needs the internet, set network = true on it; the receipt then says "internet allowed".

Does an LLM judge the work? No. The agent never grades itself, and Countercheck doesn't ask a model either. The checks you lock decide. The only text reading is a small, fixed list of phrases that tells whether the agent's final message already admits a blocker ("this can't pass, because …"), so the hook doesn't push an honest agent to keep going.

Will it catch every wrong "done"? Only what your checks test. In the benchmark, the false "done" claims left were cases the visible tests didn't cover. Stronger checks (golden files, coverage, budgets: docs/checks.md) catch more.

How much slower does it make my agent? About 1 second on the median benchmark run; longer when your checks are slow, because it runs them for real.

Which agents does it work with? Hermes Agent and Claude Code have hooks (Agent hooks). Snowflake's Cortex Code uses Claude Code's hook format, so the same hooks should work there (not yet tried on a real install). Any other agent, script or CI job can call countercheck verify directly.

Which operating systems? macOS, Linux and Windows 10/11 (x86_64). Windows needs Git for Windows. See Install.

Does my code leave my machine? No. Everything runs locally; receipts are files in .countercheck/receipts/. countercheck ui listens on 127.0.0.1 only, behind a random token.

Is it free? Yes, MIT-licensed. See Contribute to add check types, examples or agent hooks.

Status

v0.4 (CHANGELOG). macOS has the sandbox; Linux and Windows run checks without one (the receipt says so). Not yet: signed receipts, a per-host network allow-list, cost read from agent logs, Linux/Windows sandboxes, more agent hooks (Codex, Cursor, a GitHub Action). See docs/ROADMAP.md.

Contribute: make it fit your domain

Countercheck gets better with every field it learns about, and most of that needs no Rust: a domain is covered by a good checks.toml plus a small example. How agents fake "done" in game development, ML, dbt, video pipelines or a migration is knowledge we don't have and you do.

Each of these has a step-by-step guide in .claude/skills/, written for people and for AI coding agents alike (Claude Code loads them automatically). Read CONTRIBUTING.md for the gate, pull requests and commit sign-off (DCO).

Build

Rust 1.90+: cargo build --release → target/release/countercheck. Gate: cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace, python3 eval/validate.py. Countercheck also verifies itself: countercheck verify (checks in .countercheck/checks.toml).

Repository

• crates/countercheck-core: pure logic (check file, check types, protected paths, receipts, verdicts)
• apps/countercheck: the CLI (git, clean copy, sandbox, runner, hooks)
• examples/: one made-up project per domain, with checks that catch a planted mistake
• eval/, docs/eval/: the benchmark against Hermes Agent (tasks with hidden tests)
• .claude/skills/: step-by-step contribution guides · docs/architecture.md: how it works inside
• packaging/, install.sh, install.ps1, pyproject.toml: release packaging (Homebrew, install scripts, Python wheels)
• docs/checks.md: every check type · docs/hooks/: agent hooks · docs/decisions/: design decisions

MIT licensed.