toolbench

toolbench is a platform and CLI for building benchmarks for agentic tools and harnesses. Compose a benchmark (a task + a grading rubric) with a harness (the agent runtime), a loadout (the tools the agent is given), and a variant (the prompt + sandbox), run N trials per cell against a model, and get reach / pass@k / pass^k metrics with plots.

It is the benchmarking sibling of toolbase — the package manager and runtime for agent tools. toolbase serves the tools; toolbench measures how well an agent uses them. Together they close the loop for agentic tool and harness development.

Install

pip install toolbench                 # core framework
pip install 'toolbench[toolbase]'     # + resolve tools from toolbase loadouts

Requires Python ≥ 3.12. The CLI is available as both toolbench and the short alias tbe.

Quickstart

The example geometry benchmark (Euclidean distance + midpoint between two 2-D points) is a self-contained, dependency-free example that exercises the whole framework.

# Validate the wiring with no LLM calls or cost:
toolbench run --benchmark examples/geometry --model stub \
    --loadouts full_local --n 1 --max-cost-usd 0 --dry-run

# A real run (a few cheap trials):
toolbench run --benchmark examples/geometry --model claude-haiku-4-5 \
    --loadouts core_only,full_local --n 3 --max-cost-usd 0.50

Each run writes a directory under runs/<run_id>/ with a manifest.json, per-trial transcripts and artifacts, an aggregated summary.json / summary.txt, and headline plots (k-sweep, parallel-coordinates, per-stage breakdown).

Concepts

A benchmark lives in examples/<name>/ and is composed from four declarative axes — vary any of them on the command line to run an ablation:

A loadout source is one of:

• python: — import a module exposing TOOLS / make_tools() (the no-dependency escape hatch),

• toolbase: — resolve tools from a toolbase loadout, in-process (requires toolbench[toolbase]); served toolkit versions are recorded in the run manifest as reproducibility provenance, or

• mcp: — serve any MCP server's tools, stdio or HTTP (requires toolbench[mcp]):

  tools:
    sources:
      - toolbase: { loadout: my-loadout }
        select: [calculator__add]          # optional: ablate within the loadout
      - mcp: { url: "https://host/mcp", headers: { Authorization: "Bearer ${TOK}" } }
  

Metrics

For each (model × condition) cell over k trials:

• $\overline{\text{reach}}_k$ — mean rubric-weighted reach: how far through the task the agent gets, on average.
• $\text{pass@}k$ — probability that at least one of $k$ trials passes (best-of-$k$).
• $\text{pass}^{k}$ — probability that all $k$ trials pass (worst-of-$k$).

with bootstrap 95% confidence intervals and a metric-correlation matrix. A trial passes when it clears the rubric's pass criterion — every stage by default, or reach ≥ a pass_threshold once the rubric uses partial-credit stages. See Metrics for the exact estimators.

Commands

Run toolbench --help (or tbe --help) for the full reference.

Watching a run

Add --dashboard to run or resume to serve a live view of the run while it executes; the URL is printed at the top of the run's output:

toolbench run --benchmark examples/geometry --model claude-haiku-4-5 \
    --loadouts core_only,full_local --n 3 --max-cost-usd 0.50 --dashboard

The page shows a matrix of every trial per cell as it queues, runs and lands, the active trial's live activity (its messages and tool calls, each expandable to full input and output, over a per-tool timeline), its prompts and sandbox (files marked added / modified / deleted since the agent started), and the run's summary once it finalizes. The run's figures (progress, reach, spend, ETA) sit in a status bar along the bottom. For a run that has ended, or one started without the flag, use toolbench dashboard [run-id | dir]; given a directory of runs it shows every run in the top bar.

To run several benchmarks as one campaign, give each run the same --campaign NAME: they land together under runs/campaigns/NAME/, and toolbench dashboard --campaign NAME watches all of them, with each run a circle in the top bar to switch between. The server is read-only and binds to 127.0.0.1; on a remote machine, forward the port (ssh -L 8765:localhost:8765 <host>).

Also

• Runtimes — besides the API-driven orchestral runtime, toolbench drives the claude_code and codex CLIs directly (subscription auth, no API key), with per-turn token accounting and a filesystem sandbox. Copy-paste starting points live in harness_templates/.
• Judges — grade with the deterministic rule judge (default) or add an LLM second opinion (--judge rule+llm); the rule grade always stays authoritative. Any judge can be applied after the fact with toolbench regrade --judge ….
• Safeguards — trials that read the ground-truth answer key are quarantined (INTEGRITY_LEAK, scored 0) so a leak can't inflate the headline; every trial also gets a readable audit.txt of its full trajectory.

License

MIT

External orchestration

Use toolbench external prepare, external evaluate, or external conjure to score final artifacts without harness telemetry. See external workflows for provenance, Conjure setup and acceptance semantics.