Skip to main content

AnyBench

AnyBench turns the Git history of repositories you supply into coding problems, verifies them in Docker, runs coding models, and creates an HTML report. You do not need to know the repository's language or write benchmark cases yourself.

See the Humanize and Boltons examples for a measured run on public library commits, including reviewed datasets and model outcomes.

First run

First run: supply repositories, configure models, verify cases in Docker, then run candidates and view the report

Install Python 3.11+, Git, and Docker, then start the Docker daemon. Install AnyBench:

python -m pip install any-bench

Run it with one or more local repository paths or Git URLs:

anybench start /path/to/repository
anybench start /path/to/repo-a https://github.com/org/repo-b.git

On the first run, the wizard asks for a builder model, at least one candidate model, and an optional judge. Enter each model's API type, base URL, model ID, and credential. It checks the prerequisites and shows the maximum workload before any paid model calls. Confirm once to begin. The report path appears when the run finishes.

The builder drafts problems from up to 50 recent commits per repository. AnyBench prepares a Docker test environment, then retains only problems whose test fails before the historical fix and passes after it. Every verified problem is run once per candidate by default. Unverified problems are listed with reasons; they are never counted as solved or unsolved.

To set a smaller or larger workload:

anybench start /path/to/repo --commits 20 --max-problems 10

Interrupted sessions can resume without repeating recorded paid calls:

anybench start --resume .anybench/runs/SESSION_NAME

If a model call was interrupted with unknown billing state, AnyBench stops and asks for a new session rather than silently paying for it again.

Experimental live workspace

Install the optional localhost workspace and open it with:

python -m pip install 'any-bench[studio]'
anybench studio

Studio shows the full builder → environment → verification → candidate → evaluation → judge → report pipeline as a live whiteboard. Select a stage, problem, or attempt to inspect prompts, tools, Docker activity, logs, tests, timing, and scores. Start a guided run in the browser, or add --live to anybench start, build, validate, run, or evaluate to watch a CLI run. Existing CLI workflows work without the Studio extra.

Studio also lets you choose the candidate harness, set built-in client retries, import an AnyBench cases CSV, and view an instruction-to-tool trace with provider-reported token usage. Configure per-model token prices and an optional budget to see estimated spend and remaining budget. To use a custom CSV from the CLI, pass --dataset /path/to/cases.csv to anybench start.

While a run is active, Studio can adjust candidate concurrency, pause new work, resume, or stop gracefully. Runtime concurrency changes are marked in the report so they are not presented as fixed-load comparisons. See Studio details for controls, data visibility, and safe resume behavior.

Models and API keys

The wizard recommends an environment variable name for each credential, such as OPENAI_API_KEY. Set it in your shell before starting. You can also enter a key at a hidden prompt for the current run or save a literal api_key in the private .anybench/config.json file. Saved configs use 0600 permissions and .anybench/ is Git-ignored. Literal keys are excluded from run manifests, reports, and generated model files. To edit repositories or add, edit, and remove models later, run:

anybench configure

The supported endpoint protocols are OpenAI-compatible Chat Completions, OpenAI-compatible Responses, and Anthropic-compatible Messages. The judge is optional; verified local tests provide the main score.

How setup works

Docker is required for isolated verification and candidate execution. The builder proposes dependency setup for the repository's language and test tooling. AnyBench builds the image and allows one automated repair attempt. Image builds may download public dependencies; verification and benchmark containers have no network access. If the repository needs a private registry, external service, or another unsupported setup, the report explains why its problems were not verified. The generated Docker recipes and run files are under .anybench/runs/.

For custom images, headless harnesses, detailed metrics, and the individual build, validate, run, evaluate, and report commands, see the advanced guide.

Metadata

Release files for any-bench 0.6.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for any-bench 0.6.1
File Size Uploaded Provenance Actions
any_bench-0.6.1.tar.gz 282.0 kB With provenance Details for any_bench-0.6.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for any-bench 0.6.1
File Interpreter ABI Platform Actions
any_bench-0.6.1-py3-none-any.whl Python 3 none any Details for any_bench-0.6.1-py3-none-any.whl

Total release size: 548.1 kB

Release files / any_bench-0.6.1.tar.gz

Download URL any_bench-0.6.1.tar.gz
Size 282.0 kB
Tags Source
SHA-256 checksum
How to use checksums
fbcf82d0f91fc1eaf8accdf344c8ca2ede9493f4e78cf1f2fb3be0576d771fdd
BLAKE2b-256 checksum
How to use checksums
686bd3c60376e9c4a8d5a5a3de6e882281667205162e27ec2f948dd32d80bc56
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 28, 2026.

Transparency log

Release files / any_bench-0.6.1-py3-none-any.whl

Download URL any_bench-0.6.1-py3-none-any.whl
Size 266.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
3ebd61396a5dd80e9e24eede8cd578bb14f326698e001a00ccaa943afa05d0a4
BLAKE2b-256 checksum
How to use checksums
39ddf02b77a04fe8938d14f5e7649d2a5b5d3405ba8bf804e55973bb054eeaec
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 28, 2026.

Transparency log

Security for any-bench 0.6.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for any_bench-0.6.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 28, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page