robo-evals
Seeded, reproducible success rates for robot manipulation policies in MuJoCo, with confidence intervals and episode videos.
You register a policy (a Python callable, or a model behind an HTTP or WebSocket server), pick a task suite, and get back per-task success rates with Wilson confidence intervals, a JSON and Markdown report, and MP4 or GIF videos of the episodes you ask for. Every scene is derived from a base seed, so the same command gives the same numbers on any machine with the same MuJoCo build.
Install
Pick one of these. Each gives you the robo-evals command.
PyPI
pip install "robo-evals[video]"
The video extra adds MP4 output. Add ws for WebSocket policy servers.
Standalone executable (Linux, macOS on Apple silicon)
One file, no Python needed. The installer checks the download against the release's SHA256SUMS:
curl -fsSL https://raw.githubusercontent.com/superintelligenceco/robo-evals/main/install.sh | sh
Set ROBO_EVALS_VERSION=v0.2.0 to pin a version and ROBO_EVALS_INSTALL_DIR to change the
target directory (default ~/.local/bin). Windows users can download robo-evals-windows-x86_64.zip
from the latest release.
Container image (Linux amd64 and arm64)
The image renders offscreen with OSMesa, so videos work on any host without a GPU or a display.
Mount a directory at /out, and the reports and videos land in it:
mkdir -p out
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD/out:/out" \
ghcr.io/superintelligenceco/robo-evals:latest \
run --policy scripted --suite smoke --episodes 5 --video first --video-format mp4
# out/results/scripted/report.md, report.json, videos/reach_ep000.mp4, videos/push_ep000.mp4
Tags: :vX.Y.Z and :latest for releases, :edge for a manual build from main.
To evaluate your own policy file, mount it and pass its path. This example runs
examples/my_policy.py:
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD/out:/out" -v "$PWD/examples:/policies:ro" \
ghcr.io/superintelligenceco/robo-evals:latest \
run --policy /policies/my_policy.py:ReachPolicy --tasks reach --episodes 5
To evaluate a policy server on the host, add --network host and pass
--policy http://127.0.0.1:<port>.
Wheel from the latest release
pip install "robo-evals[video] @ https://github.com/superintelligenceco/robo-evals/releases/latest/download/robo_evals-<version>-py3-none-any.whl"
Replace <version> with the release version, for example 0.2.0. Each release also carries the
sdist (robo_evals-<version>.tar.gz) and a SHA256SUMS file.
From source
pip install "robo-evals[video] @ git+https://github.com/superintelligenceco/robo-evals"
On headless Linux outside the container, install EGL or OSMesa for videos (see Rendering and video).
Quickstart
robo-evals run --policy scripted --suite core --episodes 20 --video first
robo-evals run --policy path/to/my_policy.py:MyPolicy --suite core --episodes 20
Each run writes results/<policy>/report.json, report.md, and videos/. On a headless Linux
machine, robo-evals picks MUJOCO_GL=egl for you. Use MUJOCO_GL=osmesa if you have no GPU and
no EGL driver.
Baseline results
Two reference policies on the core suite, 50 episodes per task, base seed 0, default
randomization. Cells show the success rate, the 95% Wilson interval, and successes out of
episodes.
| Task | random |
scripted |
|---|---|---|
| reach | 0.0% [0.0, 7.1] (0/50) | 100.0% [92.9, 100.0] (50/50) |
| push | 0.0% [0.0, 7.1] (0/50) | 100.0% [92.9, 100.0] (50/50) |
| pick_place | 0.0% [0.0, 7.1] (0/50) | 100.0% [92.9, 100.0] (50/50) |
| drawer | 0.0% [0.0, 7.1] (0/50) | 100.0% [92.9, 100.0] (50/50) |
| stack | 0.0% [0.0, 7.1] (0/50) | 100.0% [92.9, 100.0] (50/50) |
| overall | 0.0% [0.0, 1.5] (0/250) | 100.0% [98.5, 100.0] (250/250) |
Produced with robo-evals 0.1.0, MuJoCo 3.14.0, Python 3.12 on Linux aarch64. Reproduce it with
examples/compare_baselines.sh. The scripted policy reads ground-truth object positions, so treat
it as an upper bound that proves each task is solvable, not as a learned result. It needs a mean
of 9 (reach) to 31 (stack) control steps to succeed.
Why it exists
Robot learning papers and model cards often report a success rate from a handful of rollouts, with no interval, no seed, and no video. Two numbers like "70%" and "80%" from 10 episodes each are not distinguishable, and you can't rerun the scenes that produced them. robo-evals makes the boring parts fixed:
- Same seed, same scenes. Each episode seed derives from the base seed, the task name, and the episode index. Adding a task or reordering the suite never changes the scenes of another task.
- Intervals, not point estimates. Every rate comes with a Wilson score interval, which stays honest at 0% and 100%.
- Evidence. Videos of the first episode, of every failure, or of every episode.
- Any policy. Anything that maps an observation to an action works, in process or over the network, so you can evaluate a large model without importing it into the harness.
Task catalog
All tasks share one scene: a table and a floating two-finger gripper driven by end-effector
position targets. Actions are [dx, dy, dz, grip] in [-1, 1]. One control step moves the target
up to 3 cm and runs 25 physics steps (20 Hz control, 2 ms timestep).
| Task | Instruction | Success predicate | Step limit |
|---|---|---|---|
reach |
move the gripper to the green target | grip point within 3 cm of the target | 60 |
push |
push the red cube into the green circle | cube center within 5 cm of the goal disc center (xy) | 150 |
pick_place |
pick up the blue cube and move it to the green target | cube center within 3 cm of a goal 8 to 20 cm above the table | 150 |
drawer |
open the drawer | drawer joint opened more than 12 cm | 120 |
stack |
stack the red cube on top of the blue cube | red cube within 2 cm (xy) and 8 mm (z) of resting on the blue cube, with the gripper open | 180 |
Suites: core runs all five tasks, smoke runs reach and push. Run robo-evals list to see
them with the randomization presets.
Domain randomization
Every random draw in an episode comes from one generator seeded with the episode seed.
| Knob | What it varies | Presets |
|---|---|---|
object_pose |
Object, goal, cabinet, and start pose ranges, scaled from 0 (fixed at range centers) to 1 | none: 0, default and wide: 1 |
friction |
A per-episode multiplier on the sliding friction of every geom | none: 1.0, default: 0.8 to 1.2, wide: 0.6 to 1.4 |
lighting |
Light position, intensity, and ambient level (render only, never physics) | none: off, default and wide: on |
Choose a preset with --randomization, or pass Randomization(...) to evaluate in Python.
Policy interface
A policy is any callable from an observation to an action:
class MyPolicy:
def reset(self, task: str, seed: int) -> None: # optional, called before every episode
...
def __call__(self, obs: dict) -> list[float]: # [dx, dy, dz, grip], each in [-1, 1]
...
grip is -1 for fully open and 1 for fully closed. The observation holds:
| Key | Shape | Meaning |
|---|---|---|
ee_pos |
(3,) | Grip point position in metres |
gripper |
(1,) | Finger opening, 1 = fully open |
| task keys | varies | goal_pos, object_pos, target_pos, handle_pos, or drawer_open |
state |
(n,) | All of the above, concatenated in order |
instruction |
str | The task's language instruction |
image |
(H, W, 3) uint8 | Front camera image, only with --image-obs |
--policy accepts random, scripted, zero, package.module:attr, path/to/file.py:attr, or
a server URL. If attr is a class or a function with no required arguments, robo-evals calls it
once to build the policy. See examples/my_policy.py.
Remote policies
Serve a model in its own process, environment, or machine, and point the harness at it:
robo-evals serve --policy my_pkg.model:load --port 8765 # or run your own server
robo-evals run --policy http://127.0.0.1:8765 --suite core --image-obs
The protocol (robo-evals/1) is small JSON over HTTP or WebSocket. Read
docs/protocol.md for the message format, and start from
examples/policy_server.py, a dependency-free server that you can
drop next to a LeRobot or OpenVLA checkpoint by replacing one function. WebSocket support needs
pip install "robo-evals[ws]".
Metrics
For each task, report.json holds:
success_rate: successes divided by episodes. An episode succeeds the first time the predicate holds, and ends there.ci_low,ci_high: the Wilson score interval at--confidence(95% by default). Withksuccesses innepisodes and normal quantilez, the center is(p + z²/2n) / (1 + z²/n)and the half-width isz·sqrt(p(1-p)/n + z²/4n²) / (1 + z²/n).mean_steps_to_success: mean control steps over successful episodes.episode_results: every episode's seed, outcome, step count, and video path.
overall pools every episode across tasks and also reports mean_task_success_rate, the
unweighted mean of the task rates. Compare runs side by side with
robo-evals compare results/a/report.json results/b/report.json.
Reproducibility
- Results are deterministic for a given robo-evals version, MuJoCo version, and platform. The test suite checks that two runs with the same seed produce identical reports and trajectories.
- MuJoCo does not promise bit-identical physics across versions or CPU architectures. Every report
records the robo-evals, MuJoCo, NumPy, and Python versions and the platform, so pin
mujocowhen you publish numbers. - The random baseline is seeded from the episode seed too, so even random rollouts replay exactly.
- Lighting randomization changes only rendered pixels. State-based policies see the same episode with lighting on or off.
Rendering and video
Videos and image observations need an OpenGL context. On Linux without a display, robo-evals sets
MUJOCO_GL=egl unless you set it yourself. If the backend is missing or broken, robo-evals logs a
warning, disables rendering, skips videos, and still scores every episode. The report states that
videos were unavailable. Image observations are the exception: they fail loudly, because a vision
policy can't run without them. MP4 output needs imageio-ffmpeg (the video extra); without it,
robo-evals writes GIFs. The container image sets MUJOCO_GL=osmesa and ships OSMesa and
imageio-ffmpeg, so both formats work there out of the box.
Python API
from robo_evals import Randomization, evaluate, load_policy
from robo_evals.report import write_reports
result = evaluate(
load_policy("scripted"),
suite="core",
episodes=20,
seed=0,
randomization=Randomization(object_pose=1.0, friction=(0.6, 1.4)),
video="failures",
video_dir="results/videos",
)
write_reports(result, "results/scripted")
print(result.overall)
Roadmap
- More tasks: peg insertion, door opening, and multi-step rearrangement with language variations.
- Camera randomization (pose and field of view) and texture randomization for vision policies.
- A rotating wrist and a 7-DoF arm scene, alongside the floating gripper.
- Additional simulator backends such as ManiSkill and Isaac Lab. None exist yet; the task and policy interfaces are kept backend-neutral so they can be added without changing reports.
- Parallel episode execution across processes.
Contributing
Read CONTRIBUTING.md to set up the project and add a task. Report security problems as described in SECURITY.md. This project follows the Contributor Covenant.
Citation
If you use robo-evals in research, cite it with the metadata in CITATION.cff.
License
Metadata
Release files for robo-evals 0.2.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| robo_evals-0.2.1.tar.gz | 499.6 kB | Without provenance | Details for robo_evals-0.2.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| robo_evals-0.2.1-py3-none-any.whl | Python 3 | none | any | Details for robo_evals-0.2.1-py3-none-any.whl |
Total release size: 539.0 kB
Release files / robo_evals-0.2.1.tar.gz
| Download URL | robo_evals-0.2.1.tar.gz |
|---|---|
| Size | 499.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
118580c52078bb8e214cefc51eb4e95d145110fab39c42bc4db2b9ff1a1c676f
|
|
BLAKE2b-256 checksum How to use checksums |
f73309334dfaa85a36aae4efc7cfe91a0b16af533daaa0734eda5617804e4a82
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.24 {"installer":{"name":"uv","version":"0.12.24","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Oracle Linux Server","version":"9.6","id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / robo_evals-0.2.1-py3-none-any.whl
| Download URL | robo_evals-0.2.1-py3-none-any.whl |
|---|---|
| Size | 39.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
a3d56d234baf76dcdc2c776eb88fa4d0cd347b5b252a0b77b2b944539fb4bfdb
|
|
BLAKE2b-256 checksum How to use checksums |
da3e86f21a85736db1328902d5be3c14714c665bfe1a24dcddb48638ad34fa78
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.24 {"installer":{"name":"uv","version":"0.12.24","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Oracle Linux Server","version":"9.6","id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for robo-evals 0.2.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.