Bench2Agent

Ask which benchmarks researchers evaluate on, how usage changes across years, and which new benchmarks other researchers adopt. Bench2Agent connects your signed-in Claude Code or Codex to tested benchmark tools and checks answers against the tool results and original paper evidence.

Usage frequency describes adoption, not benchmark quality.

Install · User guide · Releases

Get started

You need Python 3.10+ and Claude Code or Codex installed and signed in on your computer. Bench2Agent and the official AI CLI are separate installations. If needed, sign in using claude auth login or codex login first.

Then install and start the terminal conversation:

python -m pip install --upgrade bench2agent
bench2agent

Use an active Python environment; a virtual environment is recommended. On macOS/Linux, uv provides an alternative isolated installation:

uv tool install --upgrade bench2agent
bench2agent

Choose one installation method. On the first interactive launch, Bench2Agent automatically downloads the research-data bundle (about 45 MB), verifies its checksum and installs it as a data revision. Subsequent launches reuse the data. No GPU, database server, Bench2Agent account or API key is needed for the CLI-login route. Model usage follows your AI account's limits.

If both AI CLIs are signed in, choose one at the prompt. Your choice is saved. To choose explicitly:

bench2agent --connector claude
# Or:
bench2agent --connector codex

Ask a question, then a follow-up. /sources shows evidence, /chats lists saved conversations, /resume ID continues one, /new starts fresh, and Ctrl+C cancels an unfinished answer. Saved conversations keep their data revision after updates. The supported, tested terminal platform is Linux/WSL; macOS follows the same POSIX path. Native Windows CLI cancellation has not yet been validated.

API access is an explicit alternative: bench2agent --connector api prompts for the provider and a hidden, session-only API key. API billing is separate from a Claude or ChatGPT subscription.

Ask a question

Two shortened answers from the terminal with Claude Code on the 2026-10-19 data release (revision 8503cf1b1f7bee99). Model wording varies; the counts come from the tools and match the returned results.

You: What benchmarks are researchers using in robotics lately?

Bench2Agent: Among 513 parsed papers labeled robotics (ICLR 2026, ICML 2026 and NeurIPS 2025), LIBERO is used for evaluation in 82 (introducing paper), SIMPLER in 41 (introducing paper), and Meta-World in 29. CoRL, CVPR, ICCV and ECCV carry no field labels, so they are outside this field filter.

You: How has CIFAR-10 usage changed at ICLR from 2024 to 2026?

Bench2Agent: Stated evaluation use went from 179 of 1,894 parsed papers in 2024 to 197 of 3,060 in 2025 and 168 of 4,252 in 2026, or 94.5, 64.4 and 39.5 per 1,000. The count stayed roughly flat while ICLR grew; both steps are a significant fall under the within-venue test. Usage in papers without parsed full text is unknown.

You can also ask which newly introduced benchmarks other authors use, then ask for the evidence behind a result. In the standalone terminal, /sources shows the original sentences and paper links. Quotes and figures are checked against the tool results of the same turn. A number the checker cannot connect to a tool result is marked [unverified figure]: it could not be confirmed, which does not mean it is wrong.

Answers include repository, dataset or homepage links and reviewed introducing papers where available. Locations are our mapping; a repository link does not guarantee a direct data download. Missing links mean no confirmed location. Individually reviewed locations also state what was seen about the data, where and when.

Use the tools inside Claude Code or Codex

After installation, these shortcuts download missing data and open the official client with Bench2Agent's MCP tools:

bench2agent claude
# Or:
bench2agent codex

Ask the client to use Bench2Agent. In this route the client writes the answer; Bench2Agent's final-answer checks run in the bench2agent conversation interface. MCP setup and scripted queries.

Data coverage

The 2026-10-19 data release contains 61 editions of 11 conferences, 88,172 parsed paper-edition observations and 9,334 benchmark/dataset entries, including 8,770 entries with reviewed introduction claims and 89 with an individually reviewed code or data location (an AI review against the introducing paper's own release sentence). Revision: 8503cf1b1f7bee99.

Counts come from explicitly stated evaluation or training use in parsed arXiv HTML, with the analyzed editions and denominator reported for each query. Missing or unparsed papers remain unknown. Field filters match labels that name a research field, each with its origin (the primary area the authors declared at submission, the topic an AAAI track states, or a derived topic); proceedings volume ids are kept but are not fields, and many editions carry no field labels at all, so a field filter excludes those papers. New introduction claims in the expanded editions are not yet reviewed, so introduction discovery is less complete than usage coverage. “New” means a reviewed claim in this corpus, not the first public release worldwide.

A data revision is one SQLite database with its manifest, identified by a hash of its content. The 2026-10-19 bundle (data format v2) needs Bench2Agent 0.5.0 or later; the 2026-10-13 bundle (format v1) still installs from its explicit URL through a one-time conversion. Each conversation and its tools and checks stay on one immutable revision. External benchmarks' actual questions, labels and images are separate downloads. Data rules · Database design

Updates and web development

Version 0.5.0 is distributed on PyPI and GitHub. Code and research data update separately:

1. Update the code with your installation method (rerun its install command).
2. Run bench2agent data install to download and activate the new data. Installed data is never replaced automatically.
3. Start a new conversation (/new) to use it. Saved conversations keep the revision they started on.
4. If you use the tools inside Claude Code or Codex, restart that client so its Bench2Agent server reads the new revision.

Version 0.4.0 cannot install the new bundle and keeps working with the 2026-10-13 data. Migration and data management.

The interface plan is terminal first, then a designed local browser workspace using the same AI CLI login, tools, research data and verifier. The browser will run on your PC; a dedicated paid web backend is not required. Its launcher and shared-history integration are planned, not available in this release. Browser interface plan.

The existing optional web/HTTP MCP modules are development prototypes. The earlier hosted visitor prototype is deferred; hosted sign-in, strong visitor isolation and public operation remain unvalidated. This release provides the local terminal and MCP. Account connections.

Code is licensed under Apache-2.0.