This release is a pre-release and may not be stable for production use.
BenchFlow
Multi-turn agent benchmarking — Scene-based lifecycle for any ACP agent
What
BenchFlow runs AI agents against benchmark tasks in sandboxed environments. Single-agent, multi-agent, and multi-round patterns share one Scene-based lifecycle.
- Any ACP agent — Gemini CLI, Claude Code, Codex, OpenCode, OpenHands, OpenClaw, Pi, or your own
- Single + multi + progressive — single-agent / multi-agent (coder + reviewer, simulated user) / multi-round with a Python
BaseUsercallback - Sandboxes — Docker locally, Daytona for parallel cloud runs, Modal for serverless/GPU-backed task environments
- Hardened verifier — defaults block BenchJack/Meerkat-style reward-hacking; tasks opt out per-feature
Install
BenchFlow's current public release is 0.5.2:
pip install --upgrade benchflow
For a uv-managed CLI install or upgrade of the public release:
uv tool install --prerelease allow --upgrade 'benchflow==0.5.2'
Use the exact benchflow==0.5.2 pin for the public CLI. The
--prerelease allow flag is currently needed for BenchFlow's pinned LiteLLM
release-candidate dependency; the exact BenchFlow pin keeps you off internal
preview builds.
Internal users who want the newest preview published from main should omit
the exact public pin:
uv tool install --prerelease allow --upgrade benchflow
That installs the latest internal preview, such as 0.5.3.dev<N>. If either
command reports Executables already exist: bench, benchflow, the machine has
old entrypoints from a previous install; rerun the same command with --force
to let uv replace them.
Requires Python 3.12+ and uv. Set DAYTONA_API_KEY for Daytona runs or configure Modal auth for Modal runs; export the relevant agent API key (GEMINI_API_KEY, ANTHROPIC_API_KEY, etc.) or run claude login / codex --login for subscription auth. Provider-prefixed models may use provider-specific credentials; Azure Foundry models use AZURE_API_KEY plus AZURE_API_ENDPOINT.
Documentation
Start with Getting started, then Concepts for the mental model. Then by goal:
| If you want to… | Read |
|---|---|
| Run an eval on an existing task | Getting started |
| Understand Rollout / Scene / Role / Verifier | Concepts |
| Author a new task | Task authoring |
| Multi-agent: coder + reviewer, simulated user, BYOS, stateful envs | Use cases |
| Multi-round single-agent (progressive disclosure, oracle access) | Progressive disclosure |
| Skill evaluation (when the artifact is a skill, not a workspace) | Skill eval |
| Understand the security model | Sandbox hardening |
| Use public vs internal preview SDK releases | Release channels |
| CLI flags + commands | CLI reference |
| Python API surface | Python API reference |
Notebooks and runnable example scripts live under docs/examples/ so examples stay versioned with the docs that explain them.
Benchmark task sources
Benchmark datasets live in external Git repos and are referenced with two fields:
# benchmarks/harvey-lab/harvey-lab-gemini-flash-lite.yaml
source:
repo: benchflow-ai/benchmarks # GitHub org/repo
path: datasets/harvey-lab/tasks # optional subpath within repo
ref: main # optional branch/tag
agent: gemini
model: gemini/gemini-3.1-flash-lite-preview
Run any benchmark via the CLI:
# From a YAML config (shipped with the repo)
bench eval create --config benchmarks/harvey-lab/harvey-lab-gemini-flash-lite.yaml
# Inline — mirrors the YAML source fields
bench eval create \
--source-repo benchflow-ai/skillsbench --source-path tasks \
--agent gemini --model gemini-3.1-flash-lite-preview --sandbox daytona --concurrency 64
Repos are cloned and cached locally under .cache/datasets/ on first use.
Downstream projects should depend on the public PyPI release by default. For internal validation before the next public release, install or lock the internal preview channel with prereleases enabled; see Release channels.
Featured
- Progressive disclosure on SWE-bench Pro — the
BaseUserabstraction drives a multi-round rollout: terse round-0 prompt → failing-test hints → full spec. 5/5 oracle on Daytona, runnable demo atdocs/examples/swebench_pro_progressive_disclosure.ipynb. See Progressive disclosure.
Research artifacts
Two runnable labs validate the security story:
labs/benchjack-sandbox-hardening/— end-to-end demo that 0.2.1+ blocks three BenchJack exploits that flip 0.2.0's reward from 0.0 to 1.0.labs/reward-hack-matrix/— full reward-hack sweep across real benchmarks comparing 0.2.0 vs 0.2.2.
Audience
- Eval researchers / paper writers → Getting started → Concepts → Use cases
- Task authors → Task authoring → Sandbox hardening
- Agent builders integrating with benchflow → Concepts → Python API reference →
benchflow.agents.registry - External benchmark adapters → Task authoring → Progressive disclosure
Contributing
PRs welcome. Open against main. CI runs ruff + tests on every PR; please run ruff check . and pytest tests/ locally first.
Release channels are documented in Release channels. In
short: merges to main publish an internal preview after CI passes, while a
matching v<version> tag publishes the public release.
License
Apache-2.0.
2 maintainers
Metadata
2 maintainers
Release files for benchflow 0.5.3.dev894
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| benchflow-0.5.3.dev894.tar.gz | 841.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| benchflow-0.5.3.dev894-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 1.3 MB
Release files / benchflow-0.5.3.dev894.tar.gz
| Download URL | benchflow-0.5.3.dev894.tar.gz |
|---|---|
| Size | 841.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
82ae37913684b6dadd73cf4140ade4d2fd8aafa06290a8c1efb35375880484d3
|
|
BLAKE2b-256 checksum How to use checksums |
1702c1f11d972c382d48db1c5a89fa935328455c24c39993f27b76a7386b279d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.11.19 {"installer":{"name":"uv","version":"0.11.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / benchflow-0.5.3.dev894-py3-none-any.whl
| Download URL | benchflow-0.5.3.dev894-py3-none-any.whl |
|---|---|
| Size | 470.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ef53aaf4f9348d532f15d5d1853b1b146bbca754800c43ed702bff9b829d3c8b
|
|
BLAKE2b-256 checksum How to use checksums |
754747ab98742d0678a9e302795099b3451ab77b89e9156501e5cd5184efe140
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.11.19 {"installer":{"name":"uv","version":"0.11.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
