worldevals 0.3.0

The Inspect Evals for robotics — a curated catalog of physical-AI benchmarks built on Inspect Robots.

🌍 WorldEvals

The Inspect Evals for robotics.

A curated catalog of physical-AI / VLA benchmarks built on Inspect Robots.

📖 Browse the catalog → worldevals.org

Inspect Robots is the framework (the "Inspect AI for robotics"). WorldEvals is the collection — but unlike Inspect Evals' monorepo, each benchmark here lives in its own repository so it owns its release cadence, dependencies, hardware notes, and leaderboard. WorldEvals is the lightweight index that ties them together: what benchmarks exist, what tasks each provides, and how to install them.

• inspect-robots list tells you what's installed.
• worldevals list tells you what exists and how to get it.

Benchmarks

Benchmark	Tasks	Tags	Status
KitchenBench — 10 bimanual kitchen-manipulation tasks	10	kitchen, bimanual, manipulation	alpha

Install & use

# Inspect Robots isn't on PyPI yet, so install it from its git tag first:
pip install "inspect-robots @ git+https://github.com/robocurve/inspect-robots@v0.3.0"
pip install "worldevals @ git+https://github.com/robocurve/worldevals"

worldevals list                 # all benchmarks
worldevals list --tag bimanual  # filter by tag
worldevals info kitchenbench    # repo, install command, task keys
worldevals tasks                # Inspect Robots tasks installed locally, by benchmark

Then install a benchmark and run it through Inspect Robots:

pip install "inspect-robots @ git+https://github.com/robocurve/inspect-robots@v0.3.0"
pip install "kitchenbench @ git+https://github.com/robocurve/kitchenbench"
inspect-robots run --task kitchenbench/pour_pasta --policy kitchen_scripted --embodiment kitchen

Backends (run on real robots)

Benchmarks are embodiment-agnostic; backend adapters supply a concrete Policy + Embodiment so a benchmark runs on real hardware or a simulator. These are their own repos too (not catalog entries):

Adapter	Policy · Embodiment	Stack
inspect-robots-yam	molmoact2 · yam_arms	MolmoAct2 on I2RT YAM bimanual arms
inspect-robots-so101	lerobot · so_arm	LeRobot policies (ACT, SmolVLA, π0, …) on SO-ARM (SO-100 / SO-101) follower arms

inspect-robots run --task kitchenbench/pour_pasta --policy molmoact2 --embodiment yam_arms

Add your benchmark

A benchmark is any repo that:

1. depends on inspect-robots,
2. defines one or more Inspect Robots Tasks, and
3. registers them via [project.entry-points."inspect_robots.tasks"] (and, if it ships a sim/embodiment or policy, inspect_robots.embodiments / inspect_robots.policies).

To list it here, add a Benchmark(...) entry to src/worldevals/catalog.py and open a PR. A test validates every entry (unique name, well-formed repo URL, ≥1 task key). See KitchenBench as the reference implementation.

Development

uv venv && uv pip install -e ".[dev]"     # inspect_robots resolved from the v0.3.0 tag
uv run pre-commit install
uv run pytest --cov                        # 100% coverage required
uv run ruff check . && uv run mypy

Citation

If you use WorldEvals in your research, please cite it:

@software{worldevals,
  author  = {Robocurve},
  title   = {WorldEvals: A curated catalog of physical-AI benchmarks},
  year    = {2026},
  url     = {https://github.com/robocurve/worldevals},
  version = {0.3.0},
  license = {MIT}
}

License

MIT