worldevals 0.3.2

A curated catalog of physical-AI benchmarks built on Inspect Robots — the Inspect Evals for robotics.

WorldEvals

A curated catalog of physical-AI / VLA benchmarks: each one installable, versioned, and runnable on real robots or simulators via Inspect Robots.

If you know Inspect Evals, this is that for robotics.

Browse the catalog → worldevals.org

Note: This project is in early development. The API may change between releases, so pin a version before depending on it.

Inspect Robots is the framework; WorldEvals is the collection. Each benchmark lives in its own repository and owns its release cadence, dependencies, hardware notes, and leaderboard. WorldEvals is the lightweight index that ties them together: what benchmarks exist, what tasks each provides, and how to install them. If you come from Inspect AI, this is the Inspect Evals of that ecosystem, minus the monorepo.

• inspect-robots list tells you what's installed.
• worldevals list tells you what exists and how to get it.

Benchmarks

Benchmark	Tasks	Tags	Status
KitchenBench: 10 bimanual kitchen-manipulation tasks	10	kitchen, bimanual, manipulation	alpha

Install & use

pip install worldevals   # pulls in inspect-robots

worldevals list                 # all benchmarks
worldevals list --tag bimanual  # filter by tag
worldevals info kitchenbench    # repo, install command, task keys
worldevals tasks                # Inspect Robots tasks installed locally, by benchmark

Then install a benchmark and run it through Inspect Robots:

pip install kitchenbench
inspect-robots run --task kitchenbench/pour_pasta --policy kitchen_scripted --embodiment kitchen

Backends (run on real robots)

Benchmarks are embodiment-agnostic; backend adapters supply a concrete Policy + Embodiment so a benchmark runs on real hardware or a simulator. These are their own repos too (not catalog entries):

Adapter	Policy · Embodiment	Stack
inspect-robots-yam	molmoact2 · yam_arms	MolmoAct2 on I2RT YAM bimanual arms
inspect-robots-so101	lerobot · so_arm	LeRobot policies (ACT, SmolVLA, π0, …) on SO-ARM (SO-100 / SO-101) follower arms

inspect-robots run --task kitchenbench/pour_pasta --policy molmoact2 --embodiment yam_arms

Add your benchmark

A benchmark is any repo that:

1. depends on inspect-robots,
2. defines one or more Inspect Robots Tasks, and
3. registers them via [project.entry-points."inspect_robots.tasks"] (and, if it ships a sim/embodiment or policy, inspect_robots.embodiments / inspect_robots.policies).

To list it here, add a Benchmark(...) entry to src/worldevals/catalog.py and open a PR. A test validates every entry (unique name, well-formed repo URL, ≥1 task key). See KitchenBench as the reference implementation.

Publishing model checkpoints

Benchmarks compare policies, and a benchmark number is only interpretable when the checkpoint behind it is documented. Every model checkpoint published to the robocurve HuggingFace org must follow the model-card standard (start from the template; the standard includes a gating checklist).

Published checkpoints are catalogued in WorldPolicies (CI-validated registry + 🤗 Collection).

Development

Dependency changes: after editing dependencies in pyproject.toml, run uv lock and commit the updated lockfile: CI installs with uv sync --locked and fails with "the lockfile needs to be updated" if you forget. Day-to-day conventions (PR-only main, the required ci-ok check, one-click releases) are documented in CLAUDE.md.

Every public module, class, and function needs a docstring, enforced by ruff D1; state the contract, don't restate the name.

uv venv && uv pip install -e ".[dev]"     # inspect_robots resolved from the v0.3.0 tag
uv run pre-commit install
uv run pytest --cov                        # 100% coverage required
uv run ruff check . && uv run mypy

Citation

If you use WorldEvals in your research, please cite it:

@software{worldevals,
  author  = {Robocurve},
  title   = {WorldEvals: A curated catalog of physical-AI benchmarks},
  year    = {2026},
  url     = {https://github.com/robocurve/worldevals},
  version = {0.3.0},
  license = {MIT}
}

License

MIT