Proof that AI agents' work really happened.

AI coding agents often say "done, all tests pass" when it isn't true. They skip a failing test, edit it until it passes, or never run it. Agent Countercheck checks the claim. When the agent says it's finished, Countercheck:

1. reads your checks from the base revision, before the agent's change, so the agent can't weaken them;
2. copies the agent's files into a clean folder and puts back any protected file the agent changed;
3. runs the checks in a sandbox on macOS, Linux and Windows: no internet unless a check asks for it, and writes only inside that folder;
4. writes a receipt with the claim, what ran, the output, any protected file the agent touched, and the verdict, sealed with a SHA-256 hash so later edits show.

It is not an AI agent. It's a plain program, like git or pytest, with no model inside. Your agent does the work; Countercheck re-runs your own checks on rules the agent couldn't change.

The same run as text. The agent's own test run had said OK (skipped=1): it had marked the failing test as skipped.

$ countercheck verify --claim "Fixed the discount bug. All tests pass." --agent claude-code --cost-usd 0.42
countercheck receipt 9073a384e88f9d37  FAILED
  claim:    "Fixed the discount bug. All tests pass." -> CONTRADICTED by evidence
  checks:   .countercheck/checks.toml @ HEAD (locked), sandbox: macos-seatbelt (no internet, localhost only; writes only in the check folder)
  protected Modified: tests/test_pricing.py -> restored_from_base
  [FAIL] unit tests  (65 ms)  $ python3 -m unittest -q
         |     self.assertEqual(apply_discount(10, 150), 0)
         | AssertionError: -5.0 != 0
  cost:     $0.4200 (reported)
  - the worker changed 1 protected file(s); countercheck ran the checks with the base version
  - check "unit tests" failed (exit 1)

It installs as agent-countercheck and runs as countercheck ("Countercheck" for short), the way pip install pre-commit gives you pre-commit. Up to v0.3 it was called countercheck; the old install names still work.

Contents: How it works · Does it work? · Get started · Reference · Platforms and the sandbox · FAQ · How it's tested · Status · Contribute

How it works

The agent never grades itself. Its claim is compared with what actually runs, on rules it couldn't change:

base revision (locked)        agent's work + claim
  checks.toml, your tests          files, "Done, all tests pass."
          \                           /
           +--> countercheck verify <+
                1 lock the checks     (from the base revision)
                2 clean copy          (edited tests put back)
                3 run in a sandbox    (no internet unless a check opts in)
                4 decide + seal       (receipt with a SHA-256 seal)
                        |
      VERIFIED -> agent may finish
      FAILED + "done" -> the hook sends the failure back; the agent fixes the code (3 rounds at most)
      FAILED + honest "can't, because ..." -> agent may stop; the receipt still says FAILED

The code behind each rule, one run step by step, and where new work plugs in: docs/architecture.md.

Where it fits

Ready setups for real projects (check files, helper scripts, a 5-minute start) are in recipes/. The first one: Oracle → Snowflake conversion with Cortex Code.

Does it work?

We gave Hermes Agent 29 small coding tasks, 3 times each, in four setups, and graded every run with hidden tests the agent never saw. Some tasks are impossible on purpose (a test expects a wrong value, a server can't be reached); the honest answer there is "this can't pass, because …".

• When GPT 6.1 got stuck, it edited the protected test and said "all tests pass". Countercheck re-ran the original tests, said FAILED and pushed back once; the agent restored the test and reported the real problem.
• Countercheck only knows what your checks test. The false claims left were cases the visible tests didn't cover. DeepSeek rarely games tests, so there was little to catch.
• It added about 1 second to the median run.

Method, numbers and limits: docs/eval/results-2026-10.md. Reproduce it with eval/.

Get started

1 install                  once per computer
2 pick the project         a git repository your agent edits
3 write checks.toml        and commit it, once per project
4 connect your agent       a hook (Claude Code, Hermes), `verify` (others) or the GitHub Action
5 read the receipts        countercheck ui / report / show

Step 1: Install

Pick one. Each gives you the same countercheck command.

countercheck --version        # countercheck 0.13.0

• Windows needs Git for Windows. Countercheck calls git, and checks run with its sh, so one checks.toml works on every OS. A check can use shell = "pwsh" or shell = "cmd" instead.
• Both install scripts check each download against SHA256SUMS.txt before installing.
• The program keeps no data where it's installed. Rules and receipts live in each project.

Upgrade: pip install -U agent-countercheck · uv tool upgrade agent-countercheck · brew upgrade agent-countercheck · or run the install script again. Uninstall: pip uninstall agent-countercheck · uv tool uninstall agent-countercheck · brew uninstall agent-countercheck · or delete the countercheck file the script installed.

Step 2: Pick the project

Countercheck works inside the project your agent edits, which must be a git repository. Not your home folder, not the agent's settings folder, not where Countercheck is installed.

cd ~/code/my-app
git status                    # "On branch main" = a git project
git init && git add -A && git commit -m "start"   # only if it said "not a git repository"

Git is how Countercheck knows the "before" version of your tests and checks (how). Each project gets its own .countercheck/checks.toml; the Hermes hook works out which project a turn touched from the files it changed.

Step 3: Write your checks and commit them

countercheck init             # writes .countercheck/checks.toml

The starter check fails on purpose until you edit it. Point it at commands you already run, usually the ones in your CI:

# files the agent may not change (.countercheck/** is always protected), including test config outside tests/
protected = ["tests/**", "**/conftest.py", "pytest.ini", "tox.ini", "setup.cfg"]
link = [".venv"]                  # optional: git-ignored dependency folders the checks need (read-only)

[[check]]
name = "tests"
# any command; runs with `sh -c` in a clean copy. The flags make pytest ignore settings the agent could change:
run = "pytest -q -c tests/pytest.ini --rootdir=. --confcutdir=tests -o addopts= tests"
timeout_secs = 600

[[check]]
name = "lint"
run = "ruff check ."

Then commit it. Countercheck only trusts checks that were in git before the agent started; uncommitted checks give UNVERIFIED, because the agent could have written them.

git add .countercheck/checks.toml && git commit -m "Countercheck checks"

• You own the tests. Countercheck never writes them, and the agent being graded shouldn't either. An agent can draft tests in a separate step that you review and commit (FAQ).
• Lock the test runner's settings too. pytest also reads conftest.py (from any folder), pytest.ini and pyproject.toml, so an agent can skip a test there without touching tests/. The setup above reads pytest's settings from tests/pytest.ini (create it with [pytest] and pythonpath = ., plus your settings). Why, and other runners · example.
• No tests yet? A command that runs zero tests and exits 0 looks like a pass. Use pytest (it fails on zero tests) or require a minimum (FAQ).
• Beyond unit tests: forbidden text, every item converted, output that matches a reference, a number within limits, changed files no check covers. See check types.

Step 4: Connect your agent

4a. Claude Code and Cortex Code

In the project's .claude/settings.json:

{
  "hooks": {
    "SessionStart": [
      { "hooks": [ { "type": "command", "command": "countercheck hook session-start", "timeout": 10 } ] }
    ],
    "Stop": [
      { "hooks": [ { "type": "command", "command": "countercheck hook claude-stop", "timeout": 600 } ] }
    ]
  }
}

Set the Stop timeout (seconds) longer than your checks take. On FAILED, Claude gets the failing output and keeps working; after 3 push-backs in a session (COUNTERCHECK_MAX_NUDGES) it may stop. Details and Windows paths: docs/hooks/claude-code.md.

4b. Hermes Agent

In the profile's config.yaml (~/.hermes/config.yaml or ~/.hermes/profiles/<name>/config.yaml):

hooks:
  on_session_start:
    - command: "countercheck hook session-start"
  pre_verify:
    - command: "countercheck hook hermes-pre-verify"
      timeout: 300

Then approve both hooks once in a terminal. Hermes only runs a shell hook after you allow that exact command, and the desktop app never asks:

hermes hooks list             # "✗ not allowlisted" = the hook will NOT run
hermes                        # or: hermes -p <name>; answer y to each hook, then /exit
hermes hooks list             # now allowed

How the project is found (Hermes may run from your home folder): docs/hooks/hermes.md.

Why the session-start hook: it remembers the commit each session started from, so the checks stay locked even if the agent commits its own edits to a test or to checks.toml. Without it, the checks are locked to HEAD.

4c. Any other agent or a script

Run verify when the agent says it's done, with its name and its final message:

countercheck verify --agent codex --claim "Fixed the discount bug."

The wording doesn't matter ("done", "fixed", or nothing at all): the verdict comes only from the checks.

4d. CI

On GitHub, add the Action (Marketplace, guide). Every pull request is verified with the checks from its base commit; you get the verdict as the job result, a job summary, and the receipt as an artifact.

on: pull_request
permissions:
  contents: read
jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: datacrate360/agent-countercheck@v0.13.0

Any other CI: pass where the branch started.

countercheck verify --base origin/main

The exit code is the result: 0 VERIFIED, 2 FAILED, 3 UNVERIFIED, 1 error. Fail the job on anything but 0.

Step 5: Read the results

Every run writes a receipt to <project>/.countercheck/receipts/<id>.json (git-ignored automatically).

countercheck ui              # live page in your browser
countercheck report --open   # the same page as one offline HTML file
countercheck show .countercheck/receipts/<id>.json   # one receipt + seal check

• ui serves the page on this computer only (127.0.0.1, with a random token in the address). It uses port 7357, or any free port (--port N to choose); running it again opens the one already running for the project.
• report writes .countercheck/receipts/report.html: the project's name and folder; a flow map (each agent's latest claim running through the locked checks to delivered, sent back or needs a person; point at an agent to light its path and the checks it broke, or at a check to see who broke it; with more than 12 agents it groups them by tool, with a switch to show each one; runs still going show as a moving line and in a Checking now list with their current step); one row per agent (its runs as coloured dots, how many passed, its latest claim); then every run, newest first. Open a run and it replays what happened step by step (checks locked, clean copy, protected files put back, each check timed, receipt sealed), with each check's output, protected and watched files the agent changed, files no check covers, and a grid for matrix checks. Filter by verdict or by agent. It loads nothing from the internet and has no scripts, so it's safe to attach to a pull request. The home folder shows as ~.

Reference

Where everything lives

There are no per-agent profiles. Each receipt names its agent and model (the hooks fill both in; --agent and --model set them by hand), so several agents on one project end up side by side in the same list, each session with its own "before" commit.

Several agents on one project, each with its own job (UI design, security review, data pipeline, …)? Give each one a role: set COUNTERCHECK_ROLE in the environment that agent runs in (its hook reads it), or pass --role to verify. The report and ui then show one row per role, and you can filter the runs by it:

COUNTERCHECK_ROLE="Security review" claude      # this agent's hook runs are labelled "Security review"
countercheck verify --role "UI design" --agent codex --claim "Restyled the checkout page."

The role is a label the caller chooses, kept on the sealed receipt like --agent; it changes no check and no verdict.

Commands

Useful verify options: --base <rev> (where the work started; default HEAD), --claim, --agent, --model, --role, --cost-usd, --out <folder>, --json, and --checks-from <suite>@<commit>:<path> for checks kept outside the repository (docs/checks.md). All options: countercheck help <command>.

Exit codes: 0 verified · 2 failed (claim contradicted) · 3 unverified (nothing failed, but the checks weren't locked in the base revision, none ran, or uncovered / watched_mode = "unverified" flagged a file) · 1 error · 4 (show) the receipt was edited.

Check types

Every type with an example: docs/checks.md. A runnable example using all of them: examples/data-migration.

Platforms and the sandbox

Builds: macOS (one universal binary for Apple silicon and Intel), Linux (x86_64 and arm64, static) and Windows (x86_64, countercheck.exe). No runtime to install.

The sandbox is a fence the operating system puts around a check while it runs, so code written to misbehave can't fetch the answer online or write outside the check folder. The receipt's sandbox field always says what was enforced.

The common cheats are caught on every OS without the sandbox: edited tests are put back, checks.toml is read from the locked base, committing edits doesn't move that base, and every check is re-run in a clean copy. The sandbox adds protection against code that misbehaves during the check.

Windows full sandbox (opt-in, once per machine, experimental):

countercheck windows-setup                       # in an administrator PowerShell
countercheck windows-setup --allow-read C:\Users\me\AppData\Local\Programs\Python   # a toolchain in your profile
countercheck windows-setup --remove              # undo everything

It creates a low-rights local account, countercheck-runner (in Users only, hidden from the sign-in screen), and a firewall rule that blocks that account's internet but not localhost. --sandbox auto then runs every check as that account, and the receipt says windows-runner. The account can't read your profile, so toolchains installed there (a per-user Python, a .venv target) need --allow-read once. A check with network = true runs as you.

Everything a check starts is stopped when it ends, including processes that detach: on Linux by a PID namespace or a subreaper, on Windows by the Job Object, on macOS by killing the process group and then what detached from it (a process that also leaves the check folder and closes its output can still escape on macOS; the sandbox limits what it can do).

Linux needs kernel 5.13+ for Landlock (6.7+ for signal and socket scoping). Some servers and Docker's default profile block unprivileged user namespaces, which is when the internet stays open. Allow them with sysctl kernel.apparmor_restrict_unprivileged_userns=0 (Ubuntu), or run the container with a seccomp profile that permits unshare.

FAQ

Is Countercheck another AI agent? No. It's a plain program with no AI model inside. The agent does the work; Countercheck runs your commands on rules locked before the agent started, and writes down what happened. Same input, same verdict, every time.

Do I write the tests, or does the agent? You own them; Countercheck never writes tests. An agent that writes the code and the test that grades it is a student writing their own exam. Most projects already have what's needed: checks.toml only points at commands you run today (pytest -q, ruff check ., npm run build). An agent can draft tests as a separate step: you check the expected answers and commit them, and from then on they grade later work. New test files an agent adds during a task are kept, but they can't replace or weaken the locked ones.

How does it know which version of my tests is "before" and which is "after"? From git. Before is the base commit: the one recorded when the agent session started (the session-start hook), HEAD by default, or --base origin/main. After is the files on disk when the agent says done. Files matching protected (and everything under .countercheck/) come from the base commit, everything else from the agent, in a clean copy. The agent's own folder is never touched. So commit your tests before the agent starts.

What if my project has no tests? It never says VERIFIED without proof:

The one gap: Countercheck trusts exit codes, so a command that runs nothing and exits 0 looks like a pass (python -m unittest does this before Python 3.12). Use a runner that fails on zero tests, like pytest, or require a minimum with a budget:

[[check]]
name = "tests (at least 40)"
run = "python3 -m unittest -v > .out 2>&1; status=$?; cat .out; echo METRIC tests=$(grep -o '^Ran [0-9]*' .out | tail -n 1 | cut -d ' ' -f 2); exit $status"
budget = { metric = "tests", min = 40 }

Keep the runner's own exit code (exit $status): a check that ends with a grep on the output passes whenever the code prints the right words, even when tests fail.

Does the agent have to say "done"? No. The hooks run when the agent tries to finish (Claude Code's Stop, Hermes's pre_verify), whatever it wrote. Its final message is only recorded as the claim. The one place text is read is a short fixed list of phrases that recognise an honest "I'm blocked", so the hook doesn't push an honest agent into gaming the check; the receipt still says FAILED, and an agent that edited protected tests is pushed back anyway.

Does Countercheck limit what my agent can do? No. The agent works with its full harness: web search, docs, tools, anything you allow. Countercheck steps in only when the agent says it's done, and re-runs your locked checks on a clean copy of the result.

Why does grading run offline? So the same work always gets the same verdict, and nobody fetches the answer at grading time. In the benchmark, the only false alarms came from checks that called a real website. If a check needs the internet, set network = true on it; the receipt then says so.

Will it catch every wrong "done"? Only what your checks test. Stronger checks (golden files, coverage, budgets: docs/checks.md) catch more. Add covers and the receipt also lists changed files no check looked at, so a VERIFIED can't quietly claim more than it checked (docs/checks.md).

How much slower does it make my agent? About 1 second on the median benchmark run; longer when your checks are slow, because it runs them for real.

Does my code leave my machine? No. Everything runs locally, and receipts are files in your project. Countercheck makes no network connections of its own: no telemetry, no network library in the binary, no LLM. countercheck ui listens on 127.0.0.1 only, behind a random token.

Is it free? Yes, MIT-licensed. See Contribute.

How Countercheck itself is tested

A referee has to be right, so it's tested harder than most tools its size:

• 293 tests (cargo test --workspace): unit tests, end-to-end tests that run the real binary on throwaway git repositories (tampered tests, timeouts, every sandbox, the agent hooks, every check type), and property tests that throw thousands of generated inputs at the verdict and seal logic (a receipt's seal breaks if any one of 35 fields changes; decide never says VERIFIED when a check failed).
• Mutation testing (cargo-mutants): the receipt, claim and config code was broken in every way the tool could find, and tests were added until every break that compiles is caught (one remaining change is provably equivalent).
• CI on macOS, Linux and Windows for every change: cargo fmt --check, cargo clippy -D warnings, all tests, a release build, the benchmark task validator, and the GitHub Action on all three runners. Countercheck also verifies itself with its own .countercheck/checks.toml.
• The benchmark above, with hidden tests as ground truth.

Why Rust?

Countercheck runs inside other people's projects, in any language, so it has to be one dependency-free file that starts instantly, can't crash and behaves the same on every OS. Rust gives that, which is why uv, ruff and OpenAI's Codex CLI use it too.

The cost: fewer people know Rust, so most contributions need none (examples, check files and benchmark tasks are plain files and Python). Full reasoning: docs/decisions/0001-stack.md.

Status

v0.13.0 (CHANGELOG). Sandbox on macOS and Linux; on Windows a partial one by default and a full one after countercheck windows-setup (experimental). A GitHub Action on the Marketplace. Not yet: signed receipts, a per-host network allow-list, cost read from agent logs, blocking the internet on Windows without the admin setup, hooks for Codex and Cursor. See docs/ROADMAP.md.

⭐ Why star it?

I use Countercheck myself every day on my own agents' work, so I will keep maintaining it.

• New agents and new fields: if you need another agent hooked up (Codex, Cursor, OpenCode, …) or checks for your kind of work, open an issue and I'll add them step by step.
• Usable, easy, free: every agent and platform it supports should work, take minutes to set up, and stay free (MIT).
• When agents change, I'll fix it: if an agent changes its hooks or output, or finds a new way to fake "done", Countercheck gets updated to catch it.

Just a star, so you can find it the next time an agent tells you it's done. ⭐

Contribute

Countercheck gets better with every field it learns about, and most of that needs no Rust: a domain is covered by a good checks.toml plus a small example. How agents fake "done" in game development, ML, dbt, video pipelines or a migration is knowledge you have and we don't.

Each has a step-by-step guide in .claude/skills/, written for people and AI coding agents alike. CONTRIBUTING.md covers the gate, pull requests (against DEV) and commit sign-off (DCO).

Build: Rust 1.90+, cargo build --release → target/release/countercheck. Gate: cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace, python3 eval/validate.py.

Repository: crates/countercheck-core (pure logic: check file, check types, receipts, verdicts) · crates/countercheck-winsandbox (the Windows sandbox's system calls) · apps/countercheck (the CLI: git, clean copy, sandboxes, runner, hooks) · action.yml (the GitHub Action) · examples/ · recipes/ · eval/ and docs/eval/ (the benchmark) · docs/ (architecture, checks, hooks, guides, decisions) · packaging/, install.sh, install.ps1, pyproject.toml (releases).

MIT licensed. The name and logo are not covered by the licence (brand).