Pre-release

This release is a pre-release and may not be stable for production use.

BenchFlow

The universal environment framework — a benchmark is just a frozen environment.

What

BenchFlow is a universal environment framework: it runs AI agents against task environments and scores them through one hardened contract. A benchmark is just a frozen environment — point BenchFlow at any of them, drive it with any ACP agent, and run single-agent, multi-agent, or multi-round patterns over the same Scene-based lifecycle.

Run any benchmark — three-layer routing runs supported frameworks natively, translates unknown formats and proves equivalence with a parity gate, or runs a bespoke harness as-is; every layer emits one scored-trajectory contract. See Run any benchmark
Any ACP agent — Gemini CLI, Claude Code, Codex, OpenCode, OpenHands, Pi, or your own
Single + multi + progressive — single-agent / multi-agent (coder + reviewer, simulated user) / multi-round with a Python BaseUser callback
Loop strategies — wrap any agent in a --loop-strategy (verify-retry, self-review); every rollout captures a per-iteration reward + token trajectory, so you can plot capability against cost (can a cheap model + loops match an expensive one at equal token spend?)
task.md tasks — one file (YAML frontmatter + prompt body) replaces the split task.toml + instruction.md layout; author with bench tasks init / check / migrate / export
Hosted environments — run external PrimeIntellect / Verifiers environments through --source-env, without converting them to BenchFlow tasks
Sandboxes — Docker locally, Apple Container on Apple Silicon Macs, Daytona for parallel cloud runs (orphaned sandboxes auto-reaped at eval start), Modal for serverless/GPU-backed task environments, and AgentCore for AWS-hosted runtimes
Hardened verifier — defaults block BenchJack/Meerkat-style reward-hacking; tasks opt out per-feature
Training-ready output — scored rollouts emit a Verifiers/ORS reward record and best-effort ATIF (trainer/atif.json) / ADP (trainer/adp.jsonl) conversions; ATIF is omitted when the trajectory is empty, and conversion errors are reported in the rollout result

Quickstart

# Install or upgrade to the latest stable BenchFlow CLI
uv tool install --python 3.12 --upgrade benchflow

# Run a benchmark: any task source, any ACP agent, any sandbox
export GEMINI_API_KEY=...            # or claude auth login / codex login for subscription auth
bench eval run \
    --source-repo benchflow-ai/skillsbench --source-path tasks \
    --agent gemini --model gemini-3.1-flash-lite-preview \
    --sandbox docker

Each run writes a per-task result.json (rewards, trajectory summary, and token usage), full events under trajectory/, and a job summary.json (pass-rate, cost, and — for looped runs — a pass@iteration convergence curve). New here? Start with Getting started, or paste the agent quickstart prompt into Claude Code / Codex / Gemini CLI and let it drive the whole thing.

Install

Install or upgrade to the latest stable release from PyPI with uv:

uv tool install --python 3.12 --upgrade benchflow

Confirm with bench --version.
BenchFlow CLI releases require Python 3.12 or newer. Keep --python 3.12 in the install command so uv does not resolve an older Python-compatible package that lacks the CLI entrypoints.
If you see Executables already exist: bench, benchflow, re-run with uv tool install --python 3.12 --upgrade --force benchflow to replace stale entrypoints from an older install.
For Daytona, Modal, or AgentCore extras, install the relevant optional package, for example uv tool install --python 3.12 --upgrade 'benchflow[sandbox-daytona]'.

Internal users wanting the newest preview from main install the internal preview channel (uv tool install --python 3.12 --prerelease allow --upgrade benchflow).

Requirements & auth. Install uv; the --python 3.12 flag lets it provision a compatible interpreter for the tool install. Set DAYTONA_API_KEY for Daytona or configure Modal auth for Modal; export an agent API key (GEMINI_API_KEY, ANTHROPIC_API_KEY, …) or use subscription auth (claude auth login / codex login). Provider-prefixed models may need provider-specific credentials; Azure Foundry uses AZURE_API_KEY + AZURE_API_ENDPOINT.

Documentation

Start with Getting started, then Concepts for the mental model. Prefer to have an AI coding agent run the whole quickstart for you? Paste the agent quickstart prompt into Claude Code, Codex CLI, or Gemini CLI. Then by goal:

If you want to…	Read
Run an eval on an existing task	Getting started
Understand how BenchFlow runs any benchmark (the three-layer model)	Run any benchmark
Have an AI agent install + run the quickstart end to end	Agent quickstart prompt
Run an agent from the public agents repo (goose, qwen-code, prime-agent, …)	Running external agents
Understand Rollout / Scene / Role / Verifier	Concepts
Author a new task	Task authoring
Author a task in the native `task.md` format	Native task.md authoring
Run a hosted PrimeIntellect / Verifiers environment	CLI reference
Multi-agent: coder + reviewer, simulated user, BYOS, stateful envs	Use cases
Multi-round single-agent (progressive disclosure, oracle access)	Progressive disclosure
Skill evaluation (when the artifact is a skill, not a workspace)	Skill eval
Contribute a trajectory capture	Trajectory upload
Understand the security model	Sandbox hardening
Use public vs internal preview SDK releases	Release channels
CLI flags + commands	CLI reference
Python API surface	Python API reference

Notebooks and runnable example scripts live under docs/examples/ so examples stay versioned with the docs that explain them.

bench agent vs bench eval adopt. bench agent list / bench agent show inspect registered AI agents (the solver programs like Claude Code or Gemini CLI). Onboarding a third-party benchmark into benchmarks/<name>/ is a separate workflow — bench eval adopt <source> scaffolds and drives the conversion, and bench eval adopt <name> --verify parity-gates it. (The legacy bench agent create|run|verify commands still work as deprecated aliases.) See the CLI reference for details.

Benchmark task sources

Benchmark datasets live in external Git repos and are referenced with two fields:

# benchmarks/harvey-lab/harvey-lab-gemini-flash-lite.yaml
source:
  repo: benchflow-ai/benchmarks    # GitHub org/repo
  path: datasets/harvey-lab/tasks  # optional subpath within repo
  ref: main                         # optional branch/tag
agent: gemini
model: gemini/gemini-3.1-flash-lite-preview

Run any benchmark via the CLI:

# From a YAML config (shipped with the repo)
bench eval run --config benchmarks/harvey-lab/harvey-lab-gemini-flash-lite.yaml

# Inline — mirrors the YAML source fields
bench eval run \
    --source-repo benchflow-ai/skillsbench --source-path tasks \
    --agent gemini --model gemini-3.1-flash-lite-preview --sandbox daytona --concurrency 64

Repos are cloned and cached locally under .cache/datasets/ on first use.

Hosted environments are another source type. Instead of a repo, pass --source-env with the environment's pinned source version to run an external PrimeIntellect / Verifiers environment on its own native harness — BenchFlow preserves the hosted identity (env_uid, hub_url) and still writes the shared rollout output contract. See the CLI reference for the full hosted-environment command shape.

Downstream projects should depend on the public PyPI release by default. For internal validation before the next public release, install or lock the internal preview channel with prereleases enabled; see Release channels.

Authoring tasks

A task is one task.md (YAML frontmatter for config + a markdown prompt body) plus environment/ and verifier/ sidecars. The bench tasks commands cover the authoring lifecycle:

bench tasks init my-task                 # scaffold a task.md package under tasks/
bench tasks check tasks/my-task          # validate (default --level structural)
bench tasks migrate legacy-task/ --remove-legacy  # convert old split packages to task.md
bench tasks export tasks/my-task out/             # write a compatibility export + loss report

See Native task.md authoring and the task standard.

Featured

Progressive disclosure on SWE-bench Pro — the BaseUser abstraction drives a multi-round rollout: terse round-0 prompt → failing-test hints → full spec. 5/5 oracle on Daytona, runnable demo at docs/examples/swebench_pro_progressive_disclosure.ipynb. See Progressive disclosure.

Audience

Eval researchers / paper writers → Getting started → Concepts → Use cases
Task authors → Task authoring → Sandbox hardening
Agent builders integrating with benchflow → Concepts → Python API reference → benchflow.agents.registry
External benchmark adapters → Task authoring → Progressive disclosure

Contributing

PRs welcome. Open against main. CI runs ruff + tests on every PR; please run ruff check . and pytest tests/ locally first.

Release channels are documented in Release channels. In short: merges to main publish an internal preview after CI passes, while a matching release tag publishes the public release.

License

Apache-2.0.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

benchflow-0.6.9.dev1920.tar.gz (1.9 MB view details)

Uploaded Aug 15, 2026 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

benchflow-0.6.9.dev1920-py3-none-any.whl (1.1 MB view details)

Uploaded Aug 15, 2026 Python 3

File details

Details for the file benchflow-0.6.9.dev1920.tar.gz.

File metadata

Download URL: benchflow-0.6.9.dev1920.tar.gz
Upload date: Aug 15, 2026
Size: 1.9 MB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: uv/0.12.5 {"installer":{"name":"uv","version":"0.12.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

File hashes

Hashes for benchflow-0.6.9.dev1920.tar.gz
Algorithm	Hash digest
SHA256	`381fbd99d5a422f40e4de889b94735045a8ee251529ac08d2301fb36ae0704a0`
MD5	`b7d8a412150c6e6a402088e4fb402dc6`
BLAKE2b-256	`1f7e34cb8fc2fee6144977aaac7b5fc62d0a7b9f14c8b218239ddb0d3f00794a`

See more details on using hashes here.

File details

Details for the file benchflow-0.6.9.dev1920-py3-none-any.whl.

File metadata

Download URL: benchflow-0.6.9.dev1920-py3-none-any.whl
Upload date: Aug 15, 2026
Size: 1.1 MB
Tags: Python 3
Uploaded using Trusted Publishing? Yes
Uploaded via: uv/0.12.5 {"installer":{"name":"uv","version":"0.12.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

File hashes

Hashes for benchflow-0.6.9.dev1920-py3-none-any.whl
Algorithm	Hash digest
SHA256	`397f17331005597cdbd4d749e77c4e4c89958b82f7ada888653737e90f4161e0`
MD5	`c707c45c39da31ae262b21e7b7ef9ac2`
BLAKE2b-256	`9da428e0695d41791f68c3ffbb404fb13e4a1778cb85b1d0650751d4e0ea4aa1`