Paper

ASI-Bench: At the Dawn of Artificial Superintelligence

arXiv:2608.17271

Figure 1: Overview of ASI-Bench. Left: B3 performance across agents. Right: scores from B1 to B4, where B1 provides full methods, B2 only the method name, B3 only the research goal and data, and B4 further adds distractors.

Overview

ASI-Bench is the first benchmark to jointly evaluate general intelligence, innovation, and autonomous execution, and the first to progressively withdraw human methodological guidance within the same research project. It tests how far an AI agent can independently select methods, conduct end-to-end research, and produce verifiable scientific results instead of merely following a human-specified procedure.

Benchmark scope	Description
Tasks	60 project-level tasks
Scientific coverage	11 domains
Information levels	Four matched B1–B4 conditions with progressively less methodological guidance
Evaluation	Expert cross-review, AI-assisted auditing, sandbox execution, and scorer validation
Evaluated systems	18 agent–model configurations

The scientific objective, input data, required artifacts, and scoring criteria remain fixed across four matched guidance conditions:

Level	Information provided	What the agent must do
B1	Scientific background, method, equations, and full procedure	Implement and execute the prescribed approach
B2	Intended method and relevant constraints, without the full procedure	Turn the method into a working research workflow
B3	Objective, data, constraints, and required outputs only	Select the method and construct and validate the workflow independently
B4	B3 plus factually correct but non-essential information	Conduct the same autonomous research while resisting distraction

These are long-horizon investigations rather than isolated questions. Across the 60 tasks, complete research trajectories involve more than 2,600 interaction turns and 2,400 execution steps, spanning over 35 hours of agent execution.

The benchmark was distilled from more than 1,300 candidate research ideas through five review rounds, over 1,100 review assignments, more than 2,000 task revisions, and over 1,500 sandbox runs. Its construction and validation involved more than 31,000 human-hours.

Resources

Resource	Link
Paper	ASI-Bench: At the Dawn of Artificial Superintelligence
Project homepage & documentation	asibench.apexin.ai
Source code	apexin-ai/ASI-Bench
Leaderboard	View official results
Dataset · seed42	Apexintelligence-AI/ASI-Bench-seed42
Dataset · seed31415	Apexintelligence-AI/ASI-Bench-seed31415
Python package	asibench on PyPI

Quick start

1. Install ASI-Bench

pip install asibench
asibench --help

Requirements: Python 3.11 or later. The default installation is lightweight and includes everything needed to download public benchmark instances. For local agent execution or task authoring, install the optional full stack:

asibench is the canonical CLI. The legacy ai4sci-bench command remains only as a backwards-compatibility alias.

pip install 'asibench[full]'

2. Download a benchmark set

Each fixed-seed dataset contains its own prompts and input data. seed31415 also publishes references for local scoring; seed42 keeps references private. Choose one seed for each run:

mkdir asi-bench-run
cd asi-bench-run

asibench task pull \
  --repo seed31415 \
  --output-dir hf_instances_seed31415/

3. Run an agent

asibench run \
  --instances-dir hf_instances_seed31415/ \
  --agent-cmd 'python my_agent.py --workspace {workspace}' \
  --sandbox linux_ns \
  --output-dir out_seed31415/

The command above evaluates all four prompt levels (b1,b2,b3,b4). Use --prompt-levels only when intentionally running a subset.

4. Score locally or submit

seed31415 can be scored locally with its public references and the scoring contracts in this GitHub checkout:

asibench score \
  --repo seed31415 \
  --results-dir out_seed31415/ \
  --instances-dir hf_instances_seed31415/ \
  --tasks-dir /path/to/ASI-Bench/tasks/

The command writes a separate local_score_seed31415.json and never overwrites the produce-only result. Local scores are reproducible but non-official.

For seed42, repeat the pull and run with separate directories, then submit for private-reference scoring:

asibench login
asibench submit --results-dir out_seed42/ \
  --benchmark-repo Apexintelligence-AI/ASI-Bench-seed42

submit creates an authenticated draft on the ASI-Bench website. Review the completeness summary and confirm the draft to enter the official scoring queue. Only seed42 result directories are accepted: the CLI checks every instance ID before building a bundle or authenticating, and rejects seed31415, unknown, or mixed-seed results. If --benchmark-repo is supplied, it must identify the official seed42 dataset. asibench score --repo seed42 is rejected before reading local inputs because seed42 GT is not public. Local benchmark runs never calculate official scores.

Run configuration

Setting	Behavior
Dataset seed	seed31415 supports public local scoring; seed42 is private-reference and Portal-scored
Downloaded instances	--instances-dir is read-only; run-specific framework metadata is written under the output directory
Prompt levels	All four levels run by default; select a subset with --prompt-levels
Timeout	--timeout defaults to 10,800 seconds and applies uniformly to every task
Produce-only reports	Unscored placeholders are never displayed as 0.0; all-unscored per-task score tables are omitted
Submission	submit uploads a draft by default; --no-upload creates a local bundle only
Local scoring	score --repo seed31415 writes a separate non-official JSON report; seed42 is rejected
Custom agents	--agent-cmd supports the none and linux_ns sandboxes
Built-in agents	Use --agent with --agent-config; compatible adapters can use Docker-based os isolation

Platform note: The examples use Linux Bash, and the linux_ns sandbox requires Linux. Windows users should run it through WSL2. PowerShell and Command Prompt use different line-continuation and JSON-escaping syntax.

See Getting Started for agent configuration, sandbox selection, and platform-specific commands.

Package and CLI compatibility

• asibench is the only published Python distribution and the canonical CLI.
• Existing Python integrations may continue to import ai4sci_bench.
• ai4sci-bench remains available as a legacy CLI alias.
• Benchmark instances are downloaded separately with asibench task pull.
• Runtime and declared-output metadata come from tasks/ or --tasks-dir.
• Retired internal orchestration commands are not part of the public CLI. Use run, report, batch-report, review, and the Portal workflows.

Built-in CLI agents — installation

Install the CLI agent you want to evaluate, then select its ASI-Bench adapter with --agent.

• New model, existing harness. When a supported harness such as Codex CLI, Claude Code, or Kimi Code adds a model, select the new model through --agent-config. If the model is served through a third-party endpoint and the adapter supports that route, also set api_base, api_key, and api_protocol (openai or anthropic). No ASI-Bench code change is normally needed while the harness accepts the model and the endpoint remains protocol compatible.
• New agent harness. A new CLI or runtime can be used immediately through --agent-cmd when it can read the prepared workspace and write result files; this generic mode supports none and linux_ns. First-class built-in integration—its own --agent name, harness-specific authentication and configuration, proxy wiring, and Docker-based os support—requires ASI-Bench maintainers to implement and test a new adapter.

Agent	Install command	Notes
Claude Code	npm install -g @anthropic-ai/claude-code	Requires Anthropic API key or local login (claude /login)
Codex CLI	npm install -g @openai/codex	Requires OpenAI API key
Kimi Code	npm install -g @moonshot-ai/kimi-code	Use the npm package. The PyPI package (kimi-cli) has different CLI arguments and is not compatible with this adapter. Requires Moonshot API key or local login (kimi /login)
CodeWhale	See CodeWhale docs	Requires DeepSeek API key by default
AntiGravity	curl -fsSL https://antigravity.google/cli/install.sh | bash	Experimental

Example with Kimi Code:

asibench run --agent kimi_code_cli \
  --agent-config '{"model": "kimi-k2.7"}' \
  --output-dir out/

Framework-managed model proxies use text-only input by default. For endpoints that accept images, set "supports_image_input": true in --agent-config. Without this flag, image attachments are replaced with a short notice before the next model request. Direct endpoints outside the framework proxy remain the responsibility of the connected agent.

Contribute a task

We welcome project-level scientific tasks from the research community. Authors can draft a task locally or in the Portal, validate it, and submit a frozen revision for review.

Quick start for task authors

# 1. Create a task scaffold
asibench task create --domain physics --name my_new_task

# 2. Edit the generated files in tasks/physics/my_new_task/:
#    - task_meta.yaml   — task metadata (public)
#    - task_eval.yaml   — evaluation and generation config during authoring
#    - task_submission.yaml — Portal-only author evidence (private; never exported)
#    - generate_gt.py   — ground-truth generator (private)
#    - custom_scorer.py — custom scoring logic, if needed
#    - prompt_b1.md … prompt_b4.md — local starter prompts for the guidance gradient

# 3. Required pre-submit validation
asibench validate --pre-submit tasks/physics/my_new_task/

# 4. Required same-model trial across B1, B2, B3, and B4
asibench difficulty-check --task physics.my_new_task

# 5. Upload an exact Draft snapshot, then review and submit it in the Portal
asibench task submit --task-dir tasks/physics/my_new_task/

Submission requirements

• A complete task package must define the scientific objective, B1–B4 prompts, agent-visible inputs and required artifacts, reproducible reference generation, evaluation gates and weighted scorers, runtime dependencies, and local-testing evidence.
• Complete validate --pre-submit and a same-model difficulty-check across B1–B4 before submission.
• Record all four scores on the 0–100 scale. B1 and B2 have no score ceiling; every B3 and B4 mean score must be strictly below 40. The CLI marks B1/B2 as RECORDED and gates the verdict only on B3/B4. --threshold may make the B3/B4 ceiling stricter but cannot raise it above 40.
• Use lowercase letters, digits, and underscores for task IDs. The canonical form is domain.task_name.
• asibench task submit --task-dir ... exact-syncs the local files to an owner-only Portal Draft. It verifies the returned relative paths and SHA-256 hashes, and prints the recoverable Draft URL if synchronization fails. It never sends the Draft for review by itself.
• On first use, asibench login opens Settings → CLI tokens. Create a PAT, copy it, and paste it into the hidden prompt; the CLI validates and saves it in ~/.asibench/credentials with mode 0600. Later submissions reuse it.
• Set ASIBENCH_NO_BROWSER=1 to print URLs without opening a browser. For CI, set ASIBENCH_SUBMIT_TOKEN; tokens are intentionally not accepted as command arguments so they do not enter shell history.
• The Portal requires local_testing_done and one finite score for each level before freezing a revision; the required difficulty verdict follows the B3/B4-below-40 policy above.
• Authors do not create repository pull requests. Administrators publish accepted tasks after review.

The Portal also provides a 15-step Guided Flow for constructing a Task entirely in the browser: asibench.apexin.ai/submit/proposals/new. Use --endpoint or ASIBENCH_SUBMIT_ENDPOINT to select another Portal.

Runnable samples from the official benchmark

The following final tasks can be used to verify the runner workflow:

Task	Domain	Description
astronomy.nbody_close_encounters	Astronomy	Close-encounter few-body scattering
math.homotopy_poly_roots	Math	Isolated complex roots via homotopy continuation

Pull and run one in produce-only mode:

asibench task pull \
  --repo seed42 \
  --tasks astronomy.nbody_close_encounters \
  --output-dir example_instances/

asibench run \
  --instances-dir example_instances/ \
  --tasks astronomy.nbody_close_encounters \
  --agent-cmd 'python my_agent.py --workspace {workspace}' \
  --sandbox linux_ns \
  --output-dir example_results/

asibench submit --results-dir example_results/

For official benchmark tasks, GitHub publishes the catalog metadata, scoring configuration, and scorer implementations for auditability. Prompts and input data come from Hugging Face. Ground-truth generators, generator settings, reference specifications, and private solver assets remain exclusively on the scoring service and are not present in this repository. seed42 reference answers are private. seed31415 reference answers are intentionally public on Hugging Face, so its GitHub scorers can produce reproducible local, non-official scores.

Public scorer code only consumes pre-generated instance/reference bundles. A formal task may therefore include an evaluator-only *_eval_runtime.py helper, but never a GT generator, reference builder, or seed-to-instance entry point. The public submission_sandbox.py is execution isolation infrastructure and contains neither task answers nor generation logic.

Five opt-in sample tasks are fully public examples. Their B1–B4 prompts, ground-truth generators, and scorer implementations or configurations are available under tasks/:

• robotics.minimum_snap_trajectory_conditioning
• chemistry.bsse_counterpoise_cbs_extrapolation
• materials.phonon_dispersion
• medicine.ethics_disclosure_diagnosis_1670
• medicine.jama_id0014_malignant_an

The examples were synchronized from apexin-ai/Agent-AI4Sci-Bench at revision 2ba9258442bf53ad6c4911957234e03e767476ad.

Dataset layout

Item	Contract
Official tasks	60 final task definitions shared by both fixed-seed datasets
Public examples	5 opt-in sample tasks with complete prompts and scoring assets
Instance layout	<output-dir>/<instance-id>/ after asibench task pull
Default selection	Unfiltered official runs exclude sample tasks
Including samples	Use --include-sample or name a sample explicitly with --tasks

Pull the two fixed-seed datasets separately:

asibench task pull --repo seed42 --output-dir hf_instances_seed42/
asibench task pull --repo seed31415 --output-dir hf_instances_seed31415/

seed42 and seed31415 are the only official Hugging Face contracts, and --repo is required. They share task schemas and declared outputs but contain different generated inputs. Matching schemas live in tasks/**/task_meta.yaml; matching public scoring contracts live in task_eval.yaml and optional custom_scorer.py files. seed31415 additionally contains public reference/ directories. Private manifests, ground-truth generators, generator settings, seed42 references, and solver assets stay on the scoring side.

The task workflow uses the grouped asibench task create, task pull, and task submit commands. See Contribute a Task for the complete authoring guide.

How scoring works (and stays fair)

• Public execution framework: agent adapters, sandboxes, instance loading, output collection, and submission packaging are auditable in this repository.
• Public local scoring: seed31415 publishes references on Hugging Face and uses the GitHub scoring configuration/custom scorers through asibench score.
• Private official scoring: seed42 never publishes references; asibench submit accepts only seed42 outputs and sends them to the website for authenticated scoring.
• Score status: seed31415 local reports are explicitly non-official; its results cannot be packaged by asibench submit, and self-reported scores are not accepted as leaderboard results.

Sandboxing & reproducibility

• Runs support task, os, and linux_ns sandboxes with pinned per-task runtimes.
• Every run records agent, model, effort, sandbox, and framework provenance.
• Source checkouts are installed editable into task environments; wheel-based runs install the matching asibench version.
• Mutable caches live under ~/.asibench/, never under site-packages.
• Namespace and Docker prerequisites are checked before execution. Missing prerequisites produce a non-zero CLI exit.

Documentation

Installation, execution, submission, task contribution, catalog, and leaderboard documentation are available on the ASI-Bench website.

License

See LICENSE.