roboeval

roboeval is a CLI-driven evaluation harness for running VLAs against simulator backends through isolated HTTP services. It provides an ActionObsSpec compatibility gate before episode execution, per-component virtual environments for dependency isolation, sharded result collection, and built-in support for LITEN-style hierarchical evaluation in which a VLM planner issues subtask instructions to a low-level VLA.

Method / Contracts

roboeval treats each VLA and simulator as an independently launched component. The orchestrator communicates with a VLA policy server and a simulator worker over HTTP/JSON, validates their declared contracts, and records episode-level results from a reproducible YAML run config.

The main contract surfaces are:

Documentation map

For a compact system overview, design rationale, supported-pair notes, tuning guidance, related systems, and decision records, see architecture, design, supported pairs, tuning, related work, and the RFC index.

Installation

For full prerequisites, platform notes, and per-component dependency details, see docs/install.md.

git clone https://github.com/KE7/roboeval.git
cd roboeval
roboeval setup pi05 libero

The setup script provisions the orchestrator plus the requested VLA and simulator environments under .venvs/.

Quickstart

roboeval setup pi05 libero
roboeval serve --vla pi05 --sim libero --headless
roboeval test --validate -c configs/libero_spatial_pi05_smoke.yaml
roboeval run -c configs/libero_spatial_pi05_smoke.yaml

serve launches the selected VLA and simulator workers. run executes the YAML configuration, including the declared VLA/simulator pair, task suite, episode count, server URLs, output directory, and optional LITEN endpoint. Additional examples are in docs/quickstart.md.

Supported VLAs and Simulators

The table describes shipped coverage. It is a support matrix, not a benchmark table; supported pairs are tested end-to-end.

Supported VLA launch names are pi05, vqbet, tdmpc2, smolvla, openvla, cosmos, groot, and internvla. Supported simulator launch names are libero, libero_pro, libero_infinity, robocasa, robotwin, aloha_gym, gym_pusht, maniskill2, and metaworld.

Current limitations

• ManiSkill2 is platform-blocked on aarch64 because the required SAPIEN 2.x wheels are x86_64-only.
• bridge_octo is platform-blocked on aarch64 by its current TensorFlow/dlimp dependency chain and does not ship in the v0.1.0 support matrix.
• Some technically expressible pairs remain capability boundaries and do not ship root configs, including RoboCasa x GR00T.

Planned features

• Multi-architecture CI matrix. aarch64 is currently the primary CI path; x86_64 execution paths exist but are not in the CI matrix.
• Additional VLAs as their checkpoints become available.
• More simulators. Community contributions are welcome; see docs/extending.md.

Extending

Extension cost. Adding a new VLA averages ~200 SLOC; adding a new simulator backend averages ~230 SLOC (across the v0.1.0 release; excludes blank lines, comments, and docstrings).

• Add a VLA by implementing a policy server with /health, /info, /reset, and /predict, then registering it with roboeval serve.
• Add a simulator by implementing a SimBackendBase backend with /init, /reset, /step, /obs, /success, and /info support through the sim worker.
• Add a new compatibility path by declaring ActionObsSpec records on both sides and adding a smoke config under configs/.

See docs/extending.md for the extension architecture and step-by-step entry points.

Citations

If you use roboeval in your research, please cite us.

@software{elmaaroufi2026roboeval,
  title   = {roboeval: A reproducible evaluation harness for Vision-Language-Action models},
  author  = {Elmaaroufi, Karim and OMAR and Seshia, Sanjit A. and Zaharia, Matei},
  version = {0.1.1},
  date    = {2026-04-29},
  url     = {https://github.com/KE7/roboeval},
  license = {BSD-3-Clause}
}

License

roboeval is released under the BSD-3-Clause License.