sciworld

Turn published papers into research environments an AI agent can actually work in.

Every paper with public code is a frozen research trajectory — problem, method, baseline, metric, result. sciworld packages one of those into an isolated workspace, hands the keys to an agent (Claude Code or Codex), and lets it try to reproduce, improve, or solve from scratch — then scores the result.

pip install sciworld

Or from source:

git clone https://github.com/TaiMingLu/sciworld.git
cd sciworld
pip install -e .

CLI

sw init                                         # one-time setup wizard
sw config [section]                             # view/edit (paths, compute, prompt, test-prompt, …)
sw select [--id ID]                             # browse papers; clones <data_dir>/envs/<id>/source/
sw info --id <arxiv_id> [--full]                # show paper overview/problem/experiments
sw research --id <arxiv_id> [flags]             # build + launch a research env
sw evaluate --id <arxiv_id> [--run NAME]        # compare run scores to paper scores

sw research flags

sw research always writes a standalone launch_<run_name>.sh as a sibling of the workspace. Normal runs execute it inside tmux/screen/nohup. With --setup-only, nothing starts — the env is prepared and you can run bash <path>/launch_<run_name>.sh yourself (inside your own job scheduler, cron, or another session manager). Re-running the script is safe.

Train / test split (anti-shortcut)

After the agent sets status.json.done=true, sciworld silently enters a test phase: copies the hidden source/eval/test/ + scripts/evaluate_test.sh into the workspace, resumes the agent with a strict "be honest, do not cheat, do not modify your method" prompt, and records the test-side scores to scoring/scores_test.json. The test phase locks method/ read-only. Nothing test-related is present in the workspace during main-phase operation.

Trajectory and token capture

Each CLI invocation writes raw JSONL events to <env_dir>/trajectory/turn_NNNN_raw.jsonl and merges a unified summary into <env_dir>/trajectory/stats.json with {input_tokens, output_tokens, num_turns, duration_s, cost_usd, tool_uses} — identical shape for Claude Code and Codex.

Python API

Every CLI command is also a Python function with the same name and matching kwargs. Required args missing → TypeError; non-zero exit → RuntimeError (no interactive prompts — notebook/script-safe).

import sciworld
from pathlib import Path

info = sciworld.research(
    id="2505.19252", agent="claude-code", mode="scratch",
    loop=True, max_loops=5,
    system_prompt=Path("~/my_prompts/strict.md").expanduser(),
    test_prompt="Run evaluate_test.sh, write reflection.md, stop.",
    setup_only=False,       # True → write launch.sh, don't launch
)
# → {"arxiv_id", "run_name", "env_dir", "source_dir", "launch_script"}

sciworld.evaluate(id="2505.19252", run=info["run_name"])
sciworld.info(id="2505.19252")
sciworld.select(id="2505.19252")
sciworld.config()                               # returns current config dict
sciworld.config(section="compute")              # opens the interactive editor
sciworld.init(force=False)                      # first-time setup wizard

Prompt kwargs (system_prompt, mode_prompt, continue_prompt, test_prompt, compute) accept either a literal str or a pathlib.Path — paths are read at call time. Use import sciworld as sw if you prefer the two-letter alias.

Where things live

Default prompts

All bundled prompts live as markdown under sciworld/prompts/ — nothing prompt-related is hardcoded in Python. Per-user overrides go into ~/.sciworld/instructions/ via sw config ….

Run any command with --help for details.