reviewgate 0.3.0

A code review gate that verifies changes by executing tests, not by judging diffs. An LLM proposes edge-case tests; a deterministic harness runs each one; a behavior is a gap only if its test runs and fails.

agentboard

A review gate that verifies changes by executing tests, not by judging diffs. An LLM proposes edge-case tests from your intent and your change. A deterministic harness runs each test against the real code in a clean checkout. A behavior is reported as a gap only if its test compiles, runs, and fails. No model is involved in the pass or fail decision.

One tool, two moments: the author runs it before pushing and fixes what is real, so the change arrives cleaner; the maintainer sees it on the pull request and decides faster. Nobody is replaced at either end.

agentboard review --target src/parser.ts --intent "handle empty input"

Why

LLM code reviewers read a diff and return an opinion. There is nothing to check the opinion against. agentboard's verdict is an executed test: it passed, it failed, it did not compile, or it timed out. Every reported gap comes with a test you can run yourself and watch fail.

Quick start

Install (Python 3.11+):

pip install reviewgate

The package is reviewgate and the command is agentboard: the project is named agentboard, but that name was already taken on PyPI, so the package ships under the gate's job description instead.

Or from source for development: pip install -e ".[dev]".

The demo runs without an API key:

agentboard demo
agentboard demo --fixed

It gates a small bundled project with one planted bug. Proposals are pre-generated, so the demo exercises the deterministic part of the pipeline: the gate finds the gap, and finds it resolved on the fixed variant. A live review needs OPENAI_API_KEY (reviewer) and, for the advisory auditor, ANTHROPIC_API_KEY.

Usage

Add an .agentboard.toml at your repo root, or skip it: the profile is auto-detected from your lockfile.

base = "main"
project = "unit"
harness_notes = "Tests already import the framework; reuse existing helpers."

Reviewing a repo you don't own? agentboard never needs to write into it. Run agentboard init --user to keep the config in your user dir (~/.config/agentboard/repos/<repo>.toml), or pass --config path.toml explicitly. The review itself also leaves the tree untouched: the board and all run artifacts default to the system temp dir.

Review a change before pushing:

agentboard review --repo . --target src/parser.ts --intent "handle empty input"

Defaults: head is your current branch, base is its fork point, and the tests file is auto-detected by four rules in order: co-located, basename-matched (with closest-path disambiguation in monorepos), the test file the reviewed diff itself touches when the diff names exactly one, and, for an explicitly named target only, the sole test file in the target's own directory. Intent can come from --intent, from --issue <url>, or from the branch's commit messages if you pass neither.

The JS toolchain runs from the target's project root, found by walking up to the nearest ancestor with a lockfile (else a package.json), so a package nested inside a larger repository works, and workspace repos still install at their top level. A preflight validates refs, files, and keys before any tokens are spent.

To review uncommitted edits instead of committed refs, pass --worktree: the diff becomes working tree vs --base (default HEAD), the sandbox executes the same on-disk state it diffed, and --intent is required since uncommitted work has no commit message to derive intent from. This is the mode a coding agent uses mid-session; it is also the default for the MCP server below.

Multi-file reviews

Add files explicitly:

agentboard review --target src/parser.ts --also src/lexer.ts --also src/ast.ts:tests/ast.test.ts

--also is repeatable. Tests are auto-detected per file; pass file:tests to override. Files with no findable tests are skipped with a note.

Or select files by the blast radius of the change:

agentboard review --target src/parser.ts --scope all --depth 2

--scope computes which files a change impacts, using code-review-graph as the graph engine. It is an optional dependency: if it is not installed, the review falls back to the explicit targets. Scopes:

Scope	Selects
changed	files changed in the diff
test-gaps	impacted files with no findable tests
all	every impacted file within --depth hops

Before proposals begin, a per-depth cost curve prints the impacted file count and test-gap count at each depth. Selections larger than --max-files (default 20) require --yes. The graph decides only which files are in scope; every selected file goes through the same gate as a single-target run.

Reviewing pull requests in CI

The same gate runs as a GitHub Action and posts its findings as a PR comment. The comment shows each confirmed gap with the failing test's source and its observed output, so a reviewer can judge the evidence in seconds; everything that passed or was already covered is collapsed. It is advisory by design: gaps never fail the build, nothing is auto-approved, and the decision stays with a human at both ends.

The workflow lives at .github/workflows/agentboard-review.yml with the comment renderer at scripts/render_pr_comment.py. It picks the first changed source file in the PR, skips quietly when a PR changes no reviewable code, and skips fork PRs entirely (repository secrets are withheld from forks, so the model key is unavailable there; that is correct, not a bug).

This repository runs it on itself: PR #1 is agentboard reviewing its own pull request, finding the demo's planted bug through three functions with an executed failing test for each.

Machine-readable output

--json-out <path> writes the run as a JSON artifact (schema_version: 1): repo, base, head, intent, targets, env_error, verdict counts, and one entry per finding with behavior, status, observed, source_file, and test_code. test_code and observed are null for skipped_covered findings, which never generated a test. Exit codes are advisory: 0 means the run completed (gaps live in the JSON, never in the exit code), 1 means it could not run. This artifact is the boundary every integration consumes; the engine itself knows nothing about pull requests.

Using from a coding agent (MCP)

The gate runs as an MCP server, so a coding agent can gate its own edits before committing them:

pip install "reviewgate[mcp]"

For Claude Code:

claude mcp add agentboard -- agentboard-mcp

For any other MCP client (Cursor, etc.), the server config is:

{ "agentboard": { "command": "agentboard-mcp" } }

This exposes one tool, review, which returns the same schema_version-1 artifact as --json-out. It defaults to --worktree mode: the diff is the working tree's uncommitted edits and the sandbox executes that same on-disk state, which is the question an agent mid-session is actually asking. intent is required: the calling agent states what its change is meant to do; nothing is derived from commit messages.

The server is the same thin adapter as the GitHub Action: it builds the CLI's own arguments and runs the same review() path, and a parity test fails if the two ever accept different flags. Verdicts stay advisory here too. The tool returns findings with their test source and observed output; it never raises on a confirmed gap, because deciding what a gap means is the calling agent's (and ultimately a human's) job.

How it works

1. Propose. An LLM reads the intent and the diff and proposes behaviors, each as a runnable test. A second-pass critic looks for gaps in that coverage.
2. Gate. Each test runs against the real code in a clean checkout. The gate is deterministic and contains no LLM. Verdicts: handled, confirmed_gap, broken_test, timed_out, skipped_covered.
3. Board. Verdicts render to an HTML review board. A run fingerprint lets any two runs be compared with one string.
4. Audit. A different model flags possible false positives on confirmed gaps. The audit is advisory and never changes a verdict.

Caching and cost

Proposing tests is the only step that costs tokens, so it is the step that is cached. Proposals are keyed by intent, diff, and target; re-running an unchanged review reuses the cached set for zero tokens, and the cache id is printed when that happens. The gate always re-runs, since executing tests is cheap and re-verifying is the point.

The gate is batched: one harness invocation gates many behaviors, with a serial fallback per behavior where batching cannot isolate a result. Batched and serial paths are asserted verdict-identical by fingerprint. With --scope, the cost curve prints before any spend.

Local and open-weight models

Model routing is one rule: a model named claude* uses Anthropic; every other name uses an OpenAI-compatible client. Setting OPENAI_BASE_URL points that client anywhere, so the same install runs against a local server or a hosted open-weight provider with no code change:

# Ollama (free, offline; no key needed)
export OPENAI_BASE_URL=http://localhost:11434/v1

# or a hosted provider (OpenRouter shown; needs its key)
export OPENAI_BASE_URL=https://openrouter.ai/api/v1
export OPENAI_API_KEY=sk-or-...

# .agentboard.toml
reviewer_model = "qwen3.6:27b"            # or "moonshotai/kimi-k2.6", etc.
critic_model = "devstral-small-2"         # a different lineage decorrelates

The design absorbs weaker proposers safely: a proposal that does not compile or run is scored against the test (broken_test), never against the code, so a smaller model can only cost recall, not trust. The verdict path is unchanged because it never contained a model.

Two notes. With OPENAI_BASE_URL set, preflight cannot know whether the endpoint requires auth (Ollama ignores keys; hosted providers need one), so a missing provider key surfaces as a [warn] at propose time rather than a preflight stop. And the advisory auditor is a claude* model by default; point --audit-model at any open model, or --no-audit to skip it.

Both lanes are verified working: Kimi K2.6 via OpenRouter and local models via Ollama, reviewing the same planted bug through the same gate. Measured rows (catches, wrong-assertion false positives, cost per review) live in notes/model-comparison.md. OpenRouter gotchas learned the hard way: its keys start sk-or-v1- (an OpenAI sk-proj- key fails as "missing authentication"), and its /models endpoint answers without auth, so verify a key against /chat/completions, not /models.

Results from real repositories

• supabase/mcp#317: proposed tests reproduced a bug on main. Composite foreign keys in list_tables returned the cartesian product of column pairs, so an N-column key reported N² pairings, most of which do not exist in the schema. The fix and the generated regression tests (self-referential, cross-schema, non-primary-unique, multi-FK, three-column) are merged into main.
• colinhacks/zod#6211: ran the proposed suite against an open fix branch (#6181). The run confirmed that fix resolves a crash across 11 shapes and surfaced one residual case involving a __proto__ path element, where bracket assignment sets the prototype instead of an own key. Verified in plain JavaScript on published zod (all five inherited names crash both formatError and treeifyError), filed as issue 6211, and fixed in PR #6212 covering both functions with regression tests, red to green with no regressions.
• unjs/ufo#360: proposed tests caught withBase/withoutBase treating / and ? as base boundaries but not #, so a fragment directly after the base path broke both operations. Reported upstream with a fix and regression tests.

Every finding above was produced by executing tests, and every one is reproducible by hand.

Benchmark

BENCHMARK.md measures the harder question: can agentboard find a real bug in code it has never seen, from an intent that does not name the bug? It runs at the parent commit of recent merged bugfix PRs, with a neutral intent, and scores against the fix.

On 12 bugs across 8 repositories, 8 rows produced a real confirmed bug and 4 were exact strict catches. One row, pointed at vueuse before a known fix, caught the fix's neighborhood and two additional bugs the PR never touched. Four rows missed and are documented in full. The benchmark also records the tool's most useful failure: the advisory auditor twice called a real strict catch a false positive, once citing the buggy line as if it were the contract, which is exactly why the verdict comes from execution and never from a model.

Every run is training data

The gate is a reward function with no model in it, so every finding is a labeled example: the inputs the proposer saw, the test it wrote, and the executed verdict. --dataset appends one JSONL row per finding to a growing corpus.

agentboard review --target src/parser.ts --intent "handle empty input" --dataset

Each row stores the proposal and the executed verdict. The honest label is ran (did the test execute), derived only from the gate's status; the advisory audit is stored alongside but never overwrites it. Collection is opt-in and append-only, writing to ~/.agentboard/dataset.jsonl by default. Existing --json-out artifacts can be backfilled, so a corpus can start from runs that predate the collector (the benchmark seeds ~260 rows on its own).

The data is not yet used in the loop; it is the substrate for the model work in ROADMAP.md (run an open model against the same benchmark, then train the proposer on gate outcomes). Rows collected from public repos are clean to keep; any future company deployment keeps its own data local and never comingled, by design.

Reliability

The classification path is checked for byte-identical verdicts across 1,000 runs per verdict class on every CI push. A falsifier test (expect(1).toBe(2)) must always classify as a real failure; if an impossible test ever reads as passing, CI fails. Batched and serial gate paths are asserted verdict-identical by fingerprint.

Limitations

• The execution trust model: reviewing a repo runs that repo's code as you. The gate copies the repo to a throwaway temp dir, but the install step executes the repo's dependency lifecycle scripts and the tests run with your OS user, your network, and your environment (minus the two model provider keys, which are scrubbed). There is no container or VM boundary yet. Only review repos you would be comfortable running npm install in. Stronger isolation is on the roadmap.
• Proposal coverage is a sampling process. The proposer reaches the topic reliably but samples which edge cases; repeated runs find overlapping but not identical sets.
• Graph scoping depends on the engine's import resolution. Repositories that route exports through barrel files (export * from chains) can under-report their radius. The cost curve shows what the graph sees before anything is spent, and --also works regardless.
• Impacted files without their own tests are gated in the nearest existing test file, which can skew results toward broken_test until test scaffolding is implemented.
• The audit pass is advisory and not yet load-bearing.
• Vitest (pnpm or npm) is the supported harness.
• Current vitest is the supported target. Very old checkouts (vitest 0.2x era) tend to fail at environment preparation for toolchain reasons that predate agentboard; the run reports this as an environment failure rather than producing verdicts.
• pnpm repos run under the version the repo's own config parses: the packageManager pin is honored when modern (>= 9), and falls back to a pinned pnpm@9 for ancient or absent pins that cannot run on current Node. A single hardcoded pin broke in both directions (old pnpm dies on Node 22; pnpm 9 rejects a pnpm 10/11 pnpm-workspace.yaml config), so the rule is "run under the version this config was written for."
• Monorepos with multiple vitest projects or per-package configs need a one-line .agentboard.toml naming the project or filter, the same way you would scope CI. Unscoped, the runner may boot an unrelated (e.g. browser) project and report an environment failure rather than a verdict.

Design invariants

1. The verifier is deterministic and external. No LLM sits in the accept or reject path.
2. Correctness comes from running the code fresh, not from memory, not from a second model agreeing, and not from a test that never made a red to green transition.
3. A second model may flag disagreement. It never votes, and conflicts surface for a human instead of being averaged away.
4. Every proposal is verified against a clean tree.