benchspec 0.0.5

Benchmark what your agent does, not what it says.

benchspec runs an agent (Claude Code, Codex, or OpenCode) against a task in a fresh sandbox, checks what it actually did in the workspace, and reports the result as a comparison: with your skill versus without, one model versus another, one harness versus another.

End-of-run summary for the in-repo hello suite: the baseline and trial columns of a real three-sample run. Rows are evals, columns are arms (baseline ran the agent bare, trial installed the skill), and every non-baseline cell shows its assertion pass rate plus the delta against the baseline in percentage points. The scoped Skill `hello` invoked line is in neither column. The same matrix lands in benchmark.md, with machine-readable artifacts alongside.

Teams pick harnesses, models, and prompts by anecdote: run it once, eyeball the transcript, trust the vibe. benchspec turns that guess into a measurement. Write the goal once, run it across the configurations you care about, and read off — in percentage points — how good each one actually is at accomplishing it.

When to use benchspec. If the only question is whether one plugin helps Claude Code, claude plugin eval ships inside Claude Code and answers it on one credential, no container. The comparison below routes the other cases.

What you need

Credentials can live in a repo-root .env. A graded run can touch up to three vendors: the agent's, the binder's, and the judge's. Or exactly one: set provider = "openrouter" on the binder, the judge, and the arms, and the whole run needs only OPENROUTER_API_KEY (see configuration.md). lint is free; analyze and run spend API calls, and run also boots sandboxes. Preflight lists every missing piece and exits before anything is spent.

Getting Started

pip install benchspec

For the microsandbox isolation opt-in, add its extra:

pip install "benchspec[microsandbox]"

Pre-1.0. The eval format and the artifact schemas are the surfaces most likely to change.

An eval is one Markdown file: a prompt, then a checklist of plain-prose claims about the workspace after the agent is done. There is no checker syntax to learn; the wording is the spec. evals/hello/greets-by-name.eval.md:

---
---

## Prompt

You are working in a workspace rooted at your current working directory.
Greet Alice by name.

## Assertions

- [ ] ./Greetings/Alice.md contains the exact line 'Hello, Alice!'
- [ ] Skill `hello` invoked
  - if: {BENCHSPEC_ARM} != {BENCHSPEC_BASELINE}
- [ ] The greeting feels warm and personable, not curt or robotic

The benchmark is a block in pyproject.toml. Arms are the report columns; the baseline is what the others are measured against:

[tool.benchspec]
default-set = "default"

[tool.benchspec.sets.default]
harness  = "claude-code"
model    = "sonnet"
baseline = "baseline"
arms = [
  { name = "baseline" },   # installs nothing
  { name = "trial" },      # setup.sh installs the skill
]

Then:

benchspec lint      # static checks on the assertions
benchspec analyze   # which assertions grade deterministically, which go to the judge
benchspec run       # every (eval × arm) in its own sandbox, graded, reported

The Skill `hello` invoked line needs a skill for the trial arm to install and a setup.sh that installs it; the quickstart writes both and takes an empty directory to that first graded report. The - if: sub-bullet keeps that line out of the baseline's rate, so the delta measures the skill (scoping).

How a run works

flowchart LR
    E["greets-by-name.eval.md<br/>prompt + assertions"] --> A1["arm: baseline<br/>fresh sandbox,<br/>setup.sh installs nothing"]
    E --> A2["arm: trial<br/>fresh sandbox,<br/>setup.sh installs the hello skill"]
    A1 --> F1["facts: files, SHAs,<br/>final message, tool calls"]
    A2 --> F2["facts"]
    F1 --> G["binder: deterministic checkers<br/>everything else: LLM judge"]
    F2 --> G
    G --> R["benchmark.md + benchmark.json<br/>meta.json + index.jsonl"]

Each (eval × arm) pair is one parametrized pytest test. A cell:

1. Boots a sandbox — a Docker container by default, or a microsandbox microVM for the opt-in — from a cached snapshot with the agent CLI already installed. The first run builds the snapshot (about a minute); later runs reuse it. benchspec sandbox:build pays that cost up front, benchspec sandbox:clean reclaims the disk.
2. Seeds the clean room — the eval's optional workspace/ files land in a fresh directory mounted at /workspace, the agent's working directory.
3. Runs setup.sh, where arms diverge: it sees $BENCHSPEC_ARM, so the baseline branch exits early and the trial branch copies the skill into place.
4. Invokes the agent on the eval's prompt.
5. Collects the facts — file tree, contents, SHA-256s, the final message, the tool calls.
6. Grades — the binder maps each assertion to a deterministic checker where it can do so without risk; the judge grades everything else from the collected evidence alone.

On either backend, nothing in the guest can write back to your checkout: setup.sh reaches the skill under test through a read-only staged copy of your repo at /project (what a git clone would contain — never .env, .git, or earlier runs' artifacts). Credential exposure differs — microsandbox injects each credential at the network boundary, Docker as a plain container environment variable the agent can read. sandbox.md has the tradeoff.

benchspec run is pytest underneath, and everything after -- goes to pytest verbatim. The normal run is benchspec run -- --count 3: three samples per cell (pytest-repeat), so every delta carries a noise band; a one-sample run is flagged in the report. -n 8 fans cells across eight sandboxes to keep it fast, and -k greets-by-name (equivalently pytest -k greets-by-name) runs one eval. The repo's own make e2e defaults to three samples and six workers through the COUNT and WORKERS variables; make e2e COUNT=1 WORKERS=1 is the quick sequential pass.

benchspec vs. the alternatives

Where each alternative is the better answer:

benchspec's own cost is the top of this README: a Docker daemon, and up to three vendors' credentials. Every run pays that back in artifacts — meta.json, index.jsonl, benchmark.json — that another tool can aggregate without knowing the directory layout.

docs/research/2026-09-13-alternatives-landscape.md has the long version: unit of evaluation, isolation, and statistics per tool.

Documentation

Development

Clone to a graded run of the in-repo suite in four steps. Steps 1 and 2 cost nothing; step 3 is the first paid call.

git clone https://github.com/theycallmeswift/benchspec && cd benchspec

1. Install. uv manages the venv; make install runs uv sync. Then the free checks:

   curl -LsSf https://astral.sh/uv/install.sh | sh   # if you don't have uv
   make install
   make test     # unit suite: no credentials, no sandbox
   

2. Confirm the suite collects. Needs no credentials and no Docker:

   make e2e EVAL_ARGS="--collect-only -q"   # 36 cells: 4 evals × 3 arms × 3 samples, then 48 through OpenRouter
   

3. Set up the credentials. make e2e runs evals/e2e/hello/ twice: four evals across three Claude Code arms, judged by Codex; then the same evals across Claude Code, Codex, and OpenCode arms with the binder, the judge, and every arm on OpenRouter. It needs claude and codex on PATH, a running Docker daemon, and four credentials in .env:

   cp .env.example .env
   

   Variable	For
   CLAUDE_CODE_OAUTH_TOKEN (from claude setup-token) or ANTHROPIC_API_KEY	the arms
   GEMINI_API_KEY	the binder
   OPENAI_API_KEY	the Codex judge
   OPENROUTER_API_KEY	the second run: binder, judge, and arms through OpenRouter

   Preflight lists every missing piece in one message and exits before anything is spent.

4. Run it. The cheapest real run is one eval, one sandbox:

   make e2e COUNT=1 WORKERS=1 EVAL_ARGS="-k greets-by-name"   # one eval, one sample, sequential
   make e2e                                                  # the whole suite, three samples, six sandboxes
   

   The first run builds the sandbox snapshot (about a minute); with the snapshot cached, the whole suite takes about a minute on six workers. The report lands in tmp/evals/iteration_01/benchmark.md.

Then make lint (ruff, ty, houserules; needs GEMINI_API_KEY) before a pull request. On Claude Code on the web, the environment's setup script does steps 1 and 3 once for every session; see sandbox.md.

Issues and pull requests are welcome at github.com/theycallmeswift/benchspec. Until CONTRIBUTING.md lands, docs/style/development.md is the code style, and make test plus make lint are the bar.

License

MIT.