ASI-Bench

A project-level benchmark for evaluating LLM agents on AI for Science.

ASI-Bench tests LLM agents on real scientific tasks across physics, chemistry, biology, earth science, materials, mathematics, and more. Each task gives an agent a scientific problem plus input data; the agent must write code and produce output artifacts, which are scored against a reference answer with tolerances — not by opinion. Every task ships a difficulty ladder of prompts (B1 → B4) that reveal progressively less of the method, so a benchmark score reflects how much an agent can do with how little guidance.

• 🌐 Website & docs: coming soon
• 🏆 Leaderboard: coming soon
• 📦 Install: pip install asibench

---

Install

pip install asibench
asibench --help

Python 3.11+. asibench is the canonical command used by all documentation and examples. The legacy ai4sci-bench entry point remains available only as a backwards-compatibility alias.

The default package is intentionally lightweight and includes everything needed to pull public Hugging Face instances. Install the optional scientific/model stack only when local agent execution or authoring needs it:

pip install 'asibench[full]'

Only one Python distribution is published: asibench. Existing Python code continues to import ai4sci_bench; names such as agent-ai4sci-bench and asi-bench are not separate packages. The distribution contains the framework code only. Benchmark instances are downloaded separately with asibench task pull, so installing the CLI does not require the Hugging Face datasets to be public. Pulling works from the wheel alone. The current run implementation still loads the matching public task catalog from tasks/ (or --tasks-dir) for runtime and declared-output metadata; HF instance manifests do not yet replace that catalog.

The public CLI intentionally excludes the retired internal orchestration commands batch-run, eval, quickeval, fullrun, rerun-flagged, and pipeline. Use run, report, batch-report, review, and the Portal-based submission flows instead.

Run the benchmark with your agent

The public repository runs agents and packages their outputs. It does not score benchmark runs locally; authenticated submissions are sent to https://asibench.apexin.ai/ for official scoring.

# 1. Create and enter a dedicated working directory
mkdir task
cd task

# 2. Pull one fixed-seed task set (prompts + input data)
asibench task pull --repo seed42 --output-dir hf_instances_seed42/

# 3. Run your agent, produce outputs, no local scoring
asibench run \
  --instances-dir hf_instances_seed42/ \
  --agent-cmd 'python my_agent.py --workspace {workspace}' \
  --sandbox linux_ns \
  --output-dir out_seed42/

# 4. Sign in so the website can identify the submitter
asibench login

# 5. Upload a draft to the ASI-Bench website for official scoring
asibench submit --results-dir out_seed42/

Run seed31415 separately with --repo seed31415, --instances-dir hf_instances_seed31415/, and a distinct results directory.

The command above is written for Linux Bash, and the linux_ns sandbox itself requires Linux. On Windows, run it inside WSL2. Bash continues a command with a trailing backslash (\); Windows PowerShell uses a backtick (`), while Command Prompt uses a caret (^). For inline JSON, Bash and PowerShell can use --agent-config '{"model":"..."}'; in Command Prompt use --agent-config "{\"model\":\"...\"}".

By default, asibench run evaluates all four prompt levels (b1,b2,b3,b4). Pass --prompt-levels explicitly only when you intentionally want a subset.

By default, submit uploads an authenticated draft to https://asibench.apexin.ai/ and returns a link. Open it, review the parsed completeness summary, and confirm to enter the website's scoring queue. Use --no-upload only when you intentionally want a local bundle.

You can plug in your own agent with --agent-cmd (any command that reads a workspace and writes output files), or use a built-in adapter (direct_llm, claude_code_cli, codex_cli, …) with --agent + --agent-config. See the Getting Started guide. File-exchange agents created with --agent-cmd support none and linux_ns sandboxes; use a compatible built-in adapter when Docker-based os isolation is required.

Agent execution timeout has one source: asibench run --timeout, which defaults to 10800 seconds. Timeout fields in task metadata are ignored, so every task in a run uses the same CLI-selected limit.

Built-in CLI agents — installation

The framework ships adapters for several CLI-based agents. Install the one you want to evaluate:

Agent	Install command	Notes
Claude Code	npm install -g @anthropic-ai/claude-code	Requires Anthropic API key or local login (claude /login)
Codex CLI	npm install -g @openai/codex	Requires OpenAI API key
Kimi Code	npm install -g @moonshot-ai/kimi-code	Use the npm package. The PyPI package (kimi-cli) has different CLI arguments and is not compatible with this adapter. Requires Moonshot API key or local login (kimi /login)
CodeWhale	See CodeWhale docs	Requires DeepSeek API key by default
AntiGravity	npm install -g @anthropic-ai/antigravity	Experimental

Example with Kimi Code:

asibench run --agent kimi_code_cli \
  --agent-config '{"model": "kimi-k2.7"}' \
  --output-dir out/

Framework-managed model proxies default to text-only input, independent of the model or provider name. If an agent tool reads a generated image, the proxy replaces the attachment with a short notice before the next model request, preventing unsupported multimodal requests from being retried until timeout. Endpoints that accept visual input can opt in with "supports_image_input": true in --agent-config. Direct endpoints that do not use a framework proxy remain the responsibility of the connected agent.

Contribute a task

Have a scientific problem that would challenge an agent? We welcome task contributions from the research community. You can draft it locally or in the Portal, then complete the required local checks before submitting a frozen revision.

Quick start for task authors

# 1. Create a task scaffold
asibench task create --domain physics --name my_new_task

# 2. Edit the generated files in tasks/physics/my_new_task/:
#    - task_meta.yaml   — task metadata (public)
#    - task_eval.yaml   — evaluation config: scoring rules, parameter space (private)
#    - generate_gt.py   — ground-truth generator (private)
#    - custom_scorer.py — custom scoring logic, if needed (private)
#    - prompt_b1.md … prompt_b4.md — difficulty-ladder prompts (public)

# 3. Required pre-submit validation
asibench validate --pre-submit tasks/physics/my_new_task/

# 4. Required same-model trial across B1, B2, B3, and B4
asibench difficulty-check --task physics.my_new_task

# 5. Open the Portal task-submission form
asibench task submit

asibench task submit does not upload files or create a draft from the terminal. By default it opens https://asibench.apexin.ai/submit/proposals/new; use --endpoint or ASIBENCH_SUBMIT_ENDPOINT to select another Portal. Task submission requires careful review of metadata, files, local-test evidence, scoring details, and author confirmations. Upload the task files and finish the complete submission flow in the browser. Set ASIBENCH_NO_BROWSER=1 to print the page URL without opening a browser. The Portal requires local_testing_done and one finite score per level before it will freeze the revision for review.

Task-author validation remains available through `validate --pre-submit` and
`difficulty-check`; benchmark `run` itself is always produce-only.

Task ids must use lowercase letters, digits, and underscores, optionally in the
canonical `domain.task_name` form used by the benchmark repository.

Task authors never create repository pull requests. After review, administrators
handle repository publication separately.

### Runnable samples from the official benchmark

The following `final` tasks are available in the official benchmark dataset
and can be used to verify the runner workflow:

| Task | Domain | Description |
|------|--------|-------------|
| `astronomy.nbody_close_encounters` | Astronomy | Close-encounter few-body scattering |
| `math.homotopy_poly_roots` | Math | Isolated complex roots via homotopy continuation |

Pull and run one in produce-only mode:

```bash
asibench task pull \
  --repo seed42 \
  --tasks astronomy.nbody_close_encounters \
  --output-dir example_instances/

asibench run \
  --instances-dir example_instances/ \
  --tasks astronomy.nbody_close_encounters \
  --agent-cmd 'python my_agent.py --workspace {workspace}' \
  --sandbox linux_ns \
  --output-dir example_results/

asibench submit --results-dir example_results/

The GitHub task catalog is metadata-only for official benchmark tasks. Prompts and input data arrive together from Hugging Face. Scoring rules and reference answers remain outside this repository; benchmark outputs are submitted to the ASI-Bench website for scoring.

Five opt-in sample tasks are intentional full public examples: their B1–B4 prompts, GT generators, and scorer implementations or scorer configurations are tracked under tasks/:

• robotics.minimum_snap_trajectory_conditioning
• chemistry.bsse_counterpoise_cbs_extrapolation
• materials.phonon_dispersion
• medicine.ethics_disclosure_diagnosis_1670
• medicine.jama_id0014_malignant_an

They were synchronized from apexin-ai/Agent-AI4Sci-Bench at revision 2ba9258442bf53ad6c4911957234e03e767476ad. Task authors should use the scaffold workflow above with their own prompts, private task_eval.yaml, and ground-truth generator for validate --pre-submit testing.

asibench task pull reads the shared tasks/<instance-id>/ layout used by both fixed-seed datasets. It materializes instances as flat <output-dir>/<instance-id>/ directories for asibench run --instances-dir. The repository includes public metadata definitions for all 60 task IDs shared by Apexintelligence-AI/ASI-Bench-seed42 and Apexintelligence-AI/ASI-Bench-seed31415, so both pulls can be resolved by the local task catalog.

The public catalog contains exactly 60 final tasks and the five full public sample tasks listed above, for 65 task definitions total. Sample tasks are valid and runnable but are excluded from an unfiltered official run by default. Use asibench list --include-sample or pass --include-sample to asibench run to include them; explicitly naming a sample with --tasks also works.

Pull the two fixed-seed datasets separately; each includes its own prompts and input data:

asibench task pull --repo seed42 --output-dir hf_instances_seed42/
asibench task pull --repo seed31415 --output-dir hf_instances_seed31415/

seed42 and seed31415 are the only official Hugging Face contracts. The --repo option is mandatory so a run can never silently select a different dataset generation. Both fixed-seed datasets use the same public task schemas and declared outputs; their generated input values differ by seed. The matching schemas are stored in this repository's tasks/**/task_meta.yaml catalog. Public instances do not need framework_task_info.json: produce-only runs use the prompt, inputs, public task metadata, and agent outputs. Instance manifests, references, and scorers remain on the private scoring side.

The task workflow is available only through the grouped asibench task create, asibench task pull, and asibench task submit commands.

For the full contribution guide, see Contribute a Task.

How scoring works (and stays fair)

• The execution framework is public and auditable — agent adapters, sandboxes, instance loading, output collection, and submission packaging live in this repository.
• Scoring happens on the ASI-Bench website. asibench login identifies the submitter, and asibench submit uploads a draft for confirmation and scoring; self-reported scores are never trusted.

Sandboxing & reproducibility

Runs can execute in isolated sandboxes (--sandbox task | os | linux_ns) with pinned per-task runtimes, and every run records provenance (agent, model, effort, sandbox, framework version) so results are reproducible and comparable.

Task environments work from both source checkouts and PyPI wheels. A source checkout is installed editable into each task environment; a wheel installation installs the same asibench version and keeps mutable caches under ~/.asibench/, never under site-packages. linux_ns and os prerequisites are checked before instances start, so an unavailable namespace or Docker daemon causes a non-zero CLI exit instead of a zero-second failed result.

Documentation

Full documentation — installation, running, submitting results, contributing tasks, the task catalog, and leaderboard methodology — lives on the website: (website coming soon)

License

See LICENSE.