benchflow 0.5.1.dev869

Multi-turn agent benchmarking with ACP — run any agent, any model, any provider.

BenchFlow

Multi-turn agent benchmarking — Scene-based lifecycle for any ACP agent

What

BenchFlow runs AI agents against benchmark tasks in sandboxed environments. Single-agent, multi-agent, and multi-round patterns share one Scene-based lifecycle.

• Any ACP agent — Gemini CLI, Claude Code, Codex, OpenCode, OpenHands, OpenClaw, Pi, or your own
• Single + multi + progressive — single-agent / multi-agent (coder + reviewer, simulated user) / multi-round with a Python BaseUser callback
• Sandboxes — Docker locally, Daytona for parallel cloud runs, Modal for serverless/GPU-backed task environments
• Hardened verifier — defaults block BenchJack/Meerkat-style reward-hacking; tasks opt out per-feature

Install

uv tool install benchflow

Requires Python 3.12+ and uv. Set DAYTONA_API_KEY for Daytona runs or configure Modal auth for Modal runs; export the relevant agent API key (GEMINI_API_KEY, ANTHROPIC_API_KEY, etc.) or run claude login / codex --login for subscription auth. Provider-prefixed models may use provider-specific credentials; Azure Foundry models use AZURE_API_KEY plus AZURE_API_ENDPOINT.

Documentation

Start with Getting started, then Concepts for the mental model. Then by goal:

If you want to…	Read
Run an eval on an existing task	Getting started
Understand Rollout / Scene / Role / Verifier	Concepts
Author a new task	Task authoring
Multi-agent: coder + reviewer, simulated user, BYOS, stateful envs	Use cases
Multi-round single-agent (progressive disclosure, oracle access)	Progressive disclosure
Skill evaluation (when the artifact is a skill, not a workspace)	Skill eval
Understand the security model	Sandbox hardening
Use public vs internal preview SDK releases	Release channels
CLI flags + commands	CLI reference
Python API surface	Python API reference

Notebooks and runnable example scripts live under docs/examples/ so examples stay versioned with the docs that explain them.

Benchmark task sources

Benchmark datasets live in external Git repos and are referenced with two fields:

# benchmarks/harvey-lab/harvey-lab-gemini-flash-lite.yaml
source:
  repo: benchflow-ai/benchmarks    # GitHub org/repo
  path: datasets/harvey-lab/tasks  # optional subpath within repo
  ref: main                         # optional branch/tag
agent: gemini
model: gemini/gemini-3.1-flash-lite-preview

Run any benchmark via the CLI:

# From a YAML config (shipped with the repo)
bench eval create --config benchmarks/harvey-lab/harvey-lab-gemini-flash-lite.yaml

# Inline — mirrors the YAML source fields
bench eval create \
    --source-repo benchflow-ai/skillsbench --source-path tasks \
    --agent gemini --model gemini-3.1-flash-lite-preview --sandbox daytona --concurrency 64

Repos are cloned and cached locally under .cache/datasets/ on first use.

Downstream projects should depend on the public PyPI release by default. For internal validation before the next public release, install or lock the internal preview channel with prereleases enabled; see Release channels.

Featured

• Progressive disclosure on SWE-bench Pro — the BaseUser abstraction drives a multi-round rollout: terse round-0 prompt → failing-test hints → full spec. 5/5 oracle on Daytona, runnable demo at docs/examples/swebench_pro_progressive_disclosure.ipynb. See Progressive disclosure.

Research artifacts

Two runnable labs validate the security story:

• labs/benchjack-sandbox-hardening/ — end-to-end demo that 0.2.1+ blocks three BenchJack exploits that flip 0.2.0's reward from 0.0 to 1.0.
• labs/reward-hack-matrix/ — full reward-hack sweep across real benchmarks comparing 0.2.0 vs 0.2.2.

Audience

• Eval researchers / paper writers → Getting started → Concepts → Use cases
• Task authors → Task authoring → Sandbox hardening
• Agent builders integrating with benchflow → Concepts → Python API reference → benchflow.agents.registry
• External benchmark adapters → Task authoring → Progressive disclosure

Contributing

PRs welcome. Open against main. CI runs ruff + tests on every PR; please run ruff check . and pytest tests/ locally first.

Release channels are documented in Release channels. In short: merges to main publish an internal preview after CI passes, while a matching v<version> tag publishes the public release.

License

Apache-2.0.