reviewgate 0.6.0

A code review gate that verifies changes by executing tests, not by judging diffs. An LLM proposes edge-case tests; a deterministic harness runs each one; a behavior is a gap only if its test runs and fails.

agentboard

A review gate that proposes edge cases and executes them before judging. It does not grade diffs and does not run your existing suite: the edge cases come from an LLM reading your intent and your change, and a deterministic harness runs each proposed test against the real code in a clean checkout. A behavior is reported as a gap only if its test compiles, runs, and fails. No model is involved in the pass or fail decision.

Two verbs, one engine. prove is the author-side verb: an agent (or you) changes code, prove tries to break it before it merges. Zero flags — it reads your dirty tree or your branch, picks targets from the diff, derives intent from your commits, and answers BROKEN (N confirmed gaps, each a failing test you can run yourself), HELD (N executed attempts, none broke it — quantified absence, never certification), or STOPPED (what happened instead, cause first). Exit codes 0/2/1, built for agent loops: write, prove, fix the failing test, prove again. review is the maintainer-side verb for pull requests and CI: same engine, always-advisory exit codes. Nobody is replaced at either end.

agentboard prove
agentboard review --target src/parser.ts --intent "handle empty input"

Why

LLM code reviewers read a diff and return an opinion. There is nothing to check the opinion against. agentboard's verdict is an executed test: it passed, it failed, it did not compile, or it timed out. Every reported gap comes with a test you can run yourself and watch fail.

Quick start

Install (Python 3.11+):

pip install reviewgate

The package is reviewgate and the command is agentboard: the project is named agentboard, but that name was already taken on PyPI, so the package ships under the gate's job description instead.

Or from source for development: pip install -e ".[dev]".

The demo runs without an API key:

agentboard demo
agentboard demo --fixed

It gates a small bundled project with one planted bug. Proposals are pre-generated, so the demo exercises the deterministic part of the pipeline: the gate finds the gap, and finds it resolved on the fixed variant. A live run needs OPENAI_API_KEY (reviewer) and, for the advisory auditor, ANTHROPIC_API_KEY — or point OPENAI_BASE_URL at Ollama and pay nobody.

Then, in any repo with uncommitted work or a feature branch:

agentboard prove

Usage

Add an .agentboard.toml at your repo root, or skip it: the profile is auto-detected from your lockfile.

base = "main"
project = "unit"
harness_notes = "Tests already import the framework; reuse existing helpers."

Reviewing a repo you don't own? agentboard never needs to write into it. Run agentboard init --user to keep the config in your user dir (~/.config/agentboard/repos/<repo>.toml), or pass --config path.toml explicitly. The review itself also leaves the tree untouched: the board and all run artifacts default to the system temp dir.

Review a change before pushing:

agentboard review --repo . --target src/parser.ts --intent "handle empty input"

Defaults: head is your current branch, base is its fork point, and the tests file is auto-detected by four rules in order: co-located, basename-matched (with closest-path disambiguation in monorepos), the test file the reviewed diff itself touches when the diff names exactly one, and, for an explicitly named target only, the sole test file in the target's own directory. Intent can come from --intent, from --issue <url>, or from the branch's commit messages if you pass neither.

The JS toolchain runs from the target's project root, found by walking up to the nearest ancestor with a lockfile (else a package.json), so a package nested inside a larger repository works, and workspace repos still install at their top level. A preflight validates refs, files, and keys before any tokens are spent.

To review uncommitted edits instead of committed refs, pass --worktree: the diff becomes working tree vs --base (default HEAD), the sandbox executes the same on-disk state it diffed, and --intent is required since uncommitted work has no commit message to derive intent from. This is the mode a coding agent uses mid-session; it is also the default for the MCP server below. One guard: on a clean checkout (a gh pr checkout, a committed branch), working-tree-vs-HEAD is a near-empty diff, and a review of nothing would still "run" on intent alone. If the worktree diff does not touch the target and no --base was given, the run aborts and names the fix, rather than gating a phantom.

Multi-file reviews

Add files explicitly:

agentboard review --target src/parser.ts --also src/lexer.ts --also src/ast.ts:tests/ast.test.ts

--also is repeatable. Tests are auto-detected per file; pass file:tests to override. Files with no findable tests are skipped with a note.

Or select files by the blast radius of the change:

agentboard review --target src/parser.ts --scope all --depth 2

--scope computes which files a change impacts, using code-review-graph as the graph engine. It is an optional dependency: if it is not installed, the review falls back to the explicit targets. Scopes:

Scope	Selects
changed	files changed in the diff
test-gaps	impacted files with no findable tests
all	every impacted file within --depth hops

Before proposals begin, a per-depth cost curve prints the impacted file count and test-gap count at each depth. Selections larger than --max-files (default 20) require --yes. The graph decides only which files are in scope; every selected file goes through the same gate as a single-target run.

Reviewing pull requests in CI

The same gate runs as a GitHub Action and posts its findings as a PR comment. The comment shows each confirmed gap with the failing test's source and its observed output, so a reviewer can judge the evidence in seconds; everything that passed or was already covered is collapsed. It is advisory by design: gaps never fail the build, nothing is auto-approved, and the decision stays with a human at both ends.

The workflow lives at .github/workflows/agentboard-review.yml with the comment renderer at scripts/render_pr_comment.py. It picks the first changed source file in the PR, skips quietly when a PR changes no reviewable code, and skips fork PRs entirely (repository secrets are withheld from forks, so the model key is unavailable there; that is correct, not a bug).

This repository runs it on itself: PR #1 is agentboard reviewing its own pull request, finding the demo's planted bug through three functions with an executed failing test for each.

Machine-readable output

--json-out <path> writes the run as a JSON artifact (schema_version: 1): repo, base, head, intent, targets, env_error, verdict counts, and one entry per finding with behavior, status, observed, source_file, and test_code. test_code and observed are null for skipped_covered findings, which never generated a test. Exit codes are advisory: 0 means the run completed (gaps live in the JSON, never in the exit code), 1 means it could not run. This artifact is the boundary every integration consumes; the engine itself knows nothing about pull requests.

Using from a coding agent (MCP)

The gate runs as an MCP server, so a coding agent can gate its own edits before committing them:

Install with pipx so the server binary lands on your PATH regardless of which environment your agent runs in:

pipx install "reviewgate[mcp]"

For Claude Code:

claude mcp add agentboard -- agentboard-mcp

If you installed into a plain venv instead of pipx, register with the absolute path, since the agent's shell will not have your venv activated:

claude mcp add agentboard -- /path/to/venv/bin/agentboard-mcp

If your agent asks what this server is before adding it, the Trust and provenance section below is a paste-ready answer.

For any other MCP client (Cursor, etc.), the server config is:

{ "agentboard": { "command": "agentboard-mcp" } }

This exposes two tools. prove is the agent's loop verb: hand it a repo (and an intent for uncommitted work), get back the artifact with a one-line verdict first — BROKEN with runnable failing tests attached, HELD with the executed-attempt count, or STOPPED with the cause. The wording is shared with the CLI by construction and cannot drift. review returns the same schema_version-1 artifact as --json-out. It defaults to --worktree mode: the diff is the working tree's uncommitted edits and the sandbox executes that same on-disk state, which is the question an agent mid-session is actually asking. intent is required: the calling agent states what its change is meant to do; nothing is derived from commit messages.

The server is the same thin adapter as the GitHub Action: it builds the CLI's own arguments and runs the same review() path, and a parity test fails if the two ever accept different flags. Verdicts stay advisory here too. The tool returns findings with their test source and observed output; it never raises on a confirmed gap, because deciding what a gap means is the calling agent's (and ultimately a human's) job.

Trust and provenance

Coding agents increasingly gate unfamiliar MCP servers behind a provenance question before registering them. That is correct behavior, and this section exists to answer it, for the agent and for you.

What this server is: a thin adapter over the same CLI. It builds the CLI's own arguments and calls the same review() and prove() paths; a parity test fails if the two ever accept different flags. There is no logic in the server that the CLI does not have.

Where the code lives: source is this repository, and the package is reviewgate on PyPI. What you install is what you can read.

What it touches: tests execute locally, in a worktree sandbox of your repository. Network calls go to exactly two places: the model provider you configured with your own key (or a local Ollama, in which case none), and the GitHub API when reviewing a pull request, to read that PR's stated intent. Nothing else is contacted, and no telemetry is sent anywhere.

What it will not do: it never auto-approves, never raises on a confirmed gap, and never blocks anything by itself. Verdicts are advisory evidence handed back to the calling agent and ultimately to a human.

If your agent asks "what is this MCP server?", this is the paste-ready answer:

agentboard-mcp is the MCP adapter for reviewgate (PyPI), source at github.com/anp0429/agentboard. It runs generated edge-case tests locally in a worktree sandbox and returns advisory verdicts; it never approves or blocks on its own. Network access is limited to the model provider key the user configured and the GitHub API for PR intent. The server is a parity-tested thin wrapper over the project's own CLI.

How it works

1. Propose. An LLM reads the intent and the diff and proposes behaviors, each as a runnable test. A second-pass critic looks for gaps in that coverage.
2. Gate. Each test runs against the real code in a clean checkout. The gate is deterministic and contains no LLM. Verdicts: handled, confirmed_gap, broken_test, timed_out, skipped_covered. JS/TS repos are gated under vitest; Python repos are gated under pytest with the same verdict taxonomy. The pytest path is newer, and was hardened the honest way: five self-review rounds and a ten-stranger-repo gauntlet (see notes/prove-birth.md and notes/gauntlet.md).
3. Board. Verdicts render to an HTML review board. A run fingerprint lets any two runs be compared with one string.
4. Precision. Two advisory layers annotate confirmed gaps without touching verdicts. A deterministic pass flags gaps that fail with the verbatim same message: real gaps fail in their own words, artifacts fail in unison, so N identical failures are reported as one suspected setup cause, not N bugs (found the hard way when nine "gaps" shared one unwrapping mistake). Separately, a different model audits each confirmed gap for wrong assertions. Neither ever changes a verdict.

What it runs on

Verdicts require executing tests, so support is defined by harnesses, not adjectives. Today: vitest (TS/JS — npm and pnpm, workspaces, .test and .spec suites, tests matched by path or by the target's exported names when filenames don't cooperate) and pytest (Python — src and flat layouts, PEP 735 dependency groups). Proven against a ten-stranger- repo launch gauntlet; the scorecard, misses included, is in notes/gauntlet.md.

One guarantee that holds on any repo in the world: prove never lies about scope. It either reaches an executed verdict, or it STOPS with the actual cause as the first line — never a silent pass, never a model's guess dressed as one. Every environment that defeats it becomes a named fix with that repo as the permanent regression test; robustness ratchets the same way recall does.

Your stack next? Open an issue with a clonable repo. A harness is one file behind three contracts: find the tests, execute a proposal, classify the raised error.

Caching and cost

Proposing tests is the only step that costs tokens, so it is the step that is cached. Proposals are keyed by intent, diff, and target; re-running an unchanged review reuses the cached set for zero tokens, and the cache id is printed when that happens. The gate always re-runs, since executing tests is cheap and re-verifying is the point.

The gate is batched: one harness invocation gates many behaviors, with a serial fallback per behavior where batching cannot isolate a result. Batched and serial paths are asserted verdict-identical by fingerprint. With --scope, the cost curve prints before any spend.

Local and open-weight models

Model routing is one rule: a model named claude* uses Anthropic; every other name uses an OpenAI-compatible client. Setting OPENAI_BASE_URL points that client anywhere, so the same install runs against a local server or a hosted open-weight provider with no code change:

# Ollama (free, offline; no key needed)
export OPENAI_BASE_URL=http://localhost:11434/v1

# or a hosted provider (OpenRouter shown; needs its key)
export OPENAI_BASE_URL=https://openrouter.ai/api/v1
export OPENAI_API_KEY=sk-or-...

# .agentboard.toml
reviewer_model = "qwen3.6:27b"            # or "moonshotai/kimi-k2.6", etc.
critic_model = "devstral-small-2"         # a different lineage decorrelates
base_url = "http://localhost:11434/v1"    # optional: pin this repo's endpoint

Because one environment variable redefines what every model name means, the run log always names the endpoint each model will talk to (reviewer qwen3.6:27b via localhost:11434), so a stray export from another session is visible instead of silent. A repo can also pin its endpoint with base_url in config (an explicitly set OPENAI_BASE_URL still wins), and preflight rejects the one unambiguous mistake up front: an OpenRouter-shaped key (sk-or-...) with no base URL set would only be refused by api.openai.com, so the run stops in two seconds with the fix instead of failing mid-review.

The design absorbs weaker proposers safely: a proposal that does not compile or run is scored against the test (broken_test), never against the code, so a smaller model can only cost recall, not trust. The verdict path is unchanged because it never contained a model.

Two notes. With OPENAI_BASE_URL set, preflight cannot know whether the endpoint requires auth (Ollama ignores keys; hosted providers need one), so a missing provider key surfaces as a [warn] at propose time rather than a preflight stop. And the advisory auditor is a claude* model by default; point --audit-model at any open model, or --no-audit to skip it.

Both lanes are verified working: Kimi K2.6 via OpenRouter and local models via Ollama, reviewing the same planted bug through the same gate. Measured rows (catches, wrong-assertion false positives, cost per review) live in notes/model-comparison.md. OpenRouter gotchas learned the hard way: its keys start sk-or-v1- (an OpenAI sk-proj- key fails as "missing authentication"), and its /models endpoint answers without auth, so verify a key against /chat/completions, not /models.

Results from real repositories

• Its own repository, first: before prove merged, it reviewed its own uncommitted diff — and across five self-review rounds found twenty-five real defects in and around its own code, including a false-positive generator inside its own verdict classifier and a hole in the fix to that bug hours later. Broken proposals went 33, 29, 0, 0, 0 as the rounds taught the proposer its own repo. The full ledger, severity- labeled, with run fingerprints: notes/prove-birth.md.

• supabase/mcp#317: proposed tests reproduced a bug on main. Composite foreign keys in list_tables returned the cartesian product of column pairs, so an N-column key reported N² pairings, most of which do not exist in the schema. The fix and the generated regression tests (self-referential, cross-schema, non-primary-unique, multi-FK, three-column) are merged into main.

• colinhacks/zod#6211: ran the proposed suite against an open fix branch (#6181). The run confirmed that fix resolves a crash across 11 shapes and surfaced one residual case involving a __proto__ path element, where bracket assignment sets the prototype instead of an own key. Verified in plain JavaScript on published zod (all five inherited names crash both formatError and treeifyError), filed as issue 6211, and fixed in PR #6212 covering both functions with regression tests, red to green with no regressions.

• unjs/ufo#360: proposed tests caught withBase/withoutBase treating / and ? as base boundaries but not #, so a fragment directly after the base path broke both operations. Reported upstream with a fix and regression tests.

• python-humanize/humanize: proposed tests for the recent number.py changes (the #328 carry fix among them) caught intword formatting every value between 10^36 and 10^100 as an enormous decillion count (a 67-digit unit multiplier) and never carrying to "1.0 googol" at the boundary. Reported upstream with two candidate behaviors for the maintainers to choose between; PR offered.

• marshmallow-code/marshmallow: proposed tests for the recent fields.py changes (#2907 among them) caught nested error_messages values shared by reference across all instances of a field class and the class default itself, via the constructor's shallow dict.update merge: one instance's customization silently rewrites every sibling and all future instances. Reported upstream; PR offered.

The humanize and marshmallow findings are the first from the pytest path: the same gate, proposing and executing pytest instead of vitest, against repositories it had never seen.

Every finding above was produced by executing tests, and every one is reproducible by hand.

Benchmark

BENCHMARK.md measures the harder question: can agentboard find a real bug in code it has never seen, from an intent that does not name the bug? It runs at the parent commit of recent merged bugfix PRs, with a neutral intent, and scores against the fix.

On 12 bugs across 8 repositories, 8 rows produced a real confirmed bug and 4 were exact strict catches. One row, pointed at vueuse before a known fix, caught the fix's neighborhood and two additional bugs the PR never touched. Four rows missed and are documented in full. The benchmark also records the tool's most useful failure: the advisory auditor twice called a real strict catch a false positive, once citing the buggy line as if it were the contract, which is exactly why the verdict comes from execution and never from a model.

Every run is training data

The gate is a reward function with no model in it, so every finding is a labeled example: the inputs the proposer saw, the test it wrote, and the executed verdict. --dataset appends one JSONL row per finding to a growing corpus.

agentboard review --target src/parser.ts --intent "handle empty input" --dataset

Each row stores the proposal and the executed verdict. The honest label is ran (did the test execute), derived only from the gate's status; the advisory audit is stored alongside but never overwrites it. Collection is opt-in and append-only, writing to ~/.agentboard/dataset.jsonl by default. Existing --json-out artifacts can be backfilled, so a corpus can start from runs that predate the collector (the benchmark seeds ~260 rows on its own).

The data is not yet used in the loop; it is the substrate for the model work in ROADMAP.md (run an open model against the same benchmark, then train the proposer on gate outcomes). Rows collected from public repos are clean to keep; any future company deployment keeps its own data local and never comingled, by design.

The collaboration-loop code that predates the gate (fix agents proposing patches, multi-model argument with executed tests as referee) lives under agentboard.experimental pending that roadmap.

Reliability

The classification path is checked for byte-identical verdicts across 1,000 runs per verdict class on every CI push. A falsifier test (expect(1).toBe(2)) must always classify as a real failure; if an impossible test ever reads as passing, CI fails. Batched and serial gate paths are asserted verdict-identical by fingerprint.

Environment failures report both output streams, labeled. stderr or stdout once hid a real install error behind a harmless package-manager warning; the stream you drop is the stream holding the cause, so neither is dropped.

Limitations

• The execution trust model: reviewing a repo runs that repo's code as you. The gate copies the repo to a throwaway temp dir, but the install step executes the repo's dependency lifecycle scripts and the tests run with your OS user, your network, and your environment (minus the two model provider keys, which are scrubbed). There is no container or VM boundary yet. Only review repos you would be comfortable running npm install in. Stronger isolation is on the roadmap.
• Proposal coverage is a sampling process. The proposer reaches the topic reliably but samples which edge cases; repeated runs find overlapping but not identical sets.
• Graph scoping depends on the engine's import resolution. Repositories that route exports through barrel files (export * from chains) can under-report their radius. The cost curve shows what the graph sees before anything is spent, and --also works regardless.
• Impacted files without their own tests are gated in the nearest existing test file, which can skew results toward broken_test until test scaffolding is implemented.
• The audit pass is advisory and not yet load-bearing.
• Vitest (pnpm or npm) is the primary harness. Python/pytest repos are supported with the same verdict taxonomy, but that path is new and experimental.
• Current vitest is the supported target. Very old checkouts (vitest 0.2x era) tend to fail at environment preparation for toolchain reasons that predate agentboard; the run reports this as an environment failure rather than producing verdicts.
• pnpm repos run under the version the repo's own config parses: the packageManager pin is honored when modern (>= 9); with no pin there, a modern pin in mise.toml or .tool-versions is honored next (pins migrate — supabase/mcp dropped packageManager for mise mid-2026); and only then does it fall back to a pinned pnpm@9. A single hardcoded pin broke in both directions (old pnpm dies on Node 22; pnpm 9 rejects a pnpm 10/11 pnpm-workspace.yaml config), so the rule is "run under the version this config was written for."
• Monorepos with multiple vitest projects usually work unscoped now (the environment probe and the proposals run beside the resolved tests file, inside whichever project owns it — zod's workspace gates unscoped). A one-line .agentboard.toml naming the project or filter remains the right call when a repo boots special environments (browser projects, custom pools), the same way you would scope CI.

Design invariants

1. The verifier is deterministic and external. No LLM sits in the accept or reject path.
2. Correctness comes from running the code fresh, not from memory, not from a second model agreeing, and not from a test that never made a red to green transition.
3. A second model may flag disagreement. It never votes, and conflicts surface for a human instead of being averaged away.
4. Every proposal is verified against a clean tree.