Bench2Agent

Find which benchmarks researchers evaluate on, how their use changes, and which new benchmarks other researchers adopt. Ask in a terminal conversation, or use Bench2Agent inside Claude Code or Codex.

Answers include usage counts, the papers covered, and original paper evidence. Usage frequency describes adoption, not benchmark quality.

Install · User guide · Releases · License

Ask a question

Two shortened answers selected after testing four English prompts through MCP with the 2026-10-07 data snapshot. Counts were cross-checked against the returned tool results; model wording can vary.

You: What benchmarks are researchers using in robotics lately?

Bench2Agent: Among 509 analysed papers labelled robotics, LIBERO appears in 80 evaluation papers (introducing paper), SIMPLER in 41 (introducing paper), and Meta-World in 29. This covers ICLR 2026, ICML 2026, and NeurIPS 2025. Unlabelled papers, including CoRL, are outside this field filter.

You: How has CIFAR-10 usage changed at ICLR from 2024 to 2026?

Bench2Agent: Its stated evaluation use fell from 94.3 to 64.8 to 38.9 per 1,000 analysed papers: 178 of 1,888 in 2024, 197 of 3,038 in 2025, and 165 of 4,237 in 2026. The counts use papers with parsed full text; usage in unparsed papers is unknown.

You can also ask which newly introduced benchmarks other authors use, then ask for the evidence behind a result. In the standalone terminal, /sources shows the original sentences and paper links. Quotes and figures are checked; anything the checker cannot verify is marked in the answer.

Answers include repository, dataset or homepage links and reviewed introducing papers where available. Locations are our mapping; a repository link does not guarantee a direct data download. Missing links mean no confirmed location.

Get started

Terminal conversation — macOS / Linux

With Python 3.10+ in your environment, install and start Bench2Agent. Choose pip or the uv alternative below:

python -m pip install bench2agent
bench2agent

uv tool install bench2agent
bench2agent

uv tool install creates a separate environment for the app. pip installs into your active Python environment; a virtual environment is recommended.

The first launch downloads the 16 MB research-data snapshot and checks its checksum automatically. Later launches reuse your installed data. This snapshot contains benchmark usage and paper evidence; external benchmarks' underlying questions, labels, and images are separate downloads.

Choose OpenAI or Anthropic at the prompt, accept or change the model, and enter your API key when asked. The key is hidden and used only for that session. Existing OPENAI_API_KEY / ANTHROPIC_API_KEY environment variables also work. API access uses your provider's API billing, separately from a ChatGPT or Claude subscription.

Once installed, just run bench2agent to return. No GPU is needed. Windows setup, saved conversations, and data updates.

Use your existing Claude Code or Codex login

Bench2Agent can supply the same data to your AI client's conversation through MCP. The client handles model access through its own login. Bench2Agent's MCP server needs no separate API key.

After installing Bench2Agent, open a conversation in your preferred client. Claude Code or Codex must already be installed and signed in:

These commands download the data if needed and open a new session with Bench2Agent tools available. Ask the client to use Bench2Agent for your benchmark question. For a fresh installation using this route, follow the complete MCP setup. Your client writes its answers; Bench2Agent's final-answer checks run in the standalone terminal interface.

Web service direction

bench2agent serve starts the existing local browser interface. A hosted conversation service is planned around the same benchmark tools, with a relational database, per-user conversations and a separate collection worker. See the database and hosting design for the terminal, web and future Mac server architecture.

Data coverage

The current snapshot covers 28 editions of 10 conferences, with 58,616 parsed paper-edition observations and 9,332 benchmark and dataset entries. Of those entries, 8,768 have reviewed introduction claims inside this corpus (review rubric).

Counts come from explicitly stated evaluation or training use in parsed arXiv HTML. Each answer reports its scope and denominator. Missing full text or an unstated role does not establish that a benchmark was unused. Field filters cover papers with matching labels and can exclude unlabelled venues.

“New” means a reviewed introduction claim inside this corpus, which starts in 2023; it does not establish the first public release. Earlier observed use, self-use, use by other authors, and uncertain attribution are recorded separately. Recent introductions have less time to accumulate adoption.

Bench2Agent is a research prototype. Released snapshots have fixed identities, and conversations record the snapshot used for their answers. Data rules and tool details.

Releases and development

Code and data are distributed separately. Bench2Agent 0.3.0 is available on PyPI and as a GitHub release. Earlier code tags retain the installation artifacts published before the rename. The data-20261007 release is the current research snapshot. Update code with uv tool upgrade bench2agent or python -m pip install --upgrade bench2agent; data updates use bench2agent data install. Existing users can follow the migration guide. See how releases work.

The earlier paper-reading interface remains available as an evidence surface: browser guide. Start it with bench2agent serve. Paper-view principles are in docs/paper-view.md.

Bench2Agent's code is licensed under Apache-2.0.