pera

The benchmarking platform for physical intelligence. This is its open SDK.

pera evaluates robot policies (VLAs) on canonical, versioned simulation protocols: same tasks, same seeds, same rules for every model. Your model stays on your GPU; the benchmark runs on pera's servers over a WebSocket Environment API. Every published score is anchored to the model authors' own harness run on the same machine, and links to per-episode videos, action traces, labeled failures, and a checksummed evidence bundle anyone can verify.

Models evaluated so far include OpenVLA-OFT, GR00T N1.6, π0.5, X-VLA, SmolVLA, Octo, MolmoAct2, and PulseVLA across the four LIBERO suites and SimplerEnv WidowX.

Install

pip install pera-sdk

Python ≥ 3.10. Runtime dependencies are only numpy and websockets — no framework, validation stack, HTTP client, or computer-vision package to fight your model's environment.

Sixty seconds to a score

import pera

MANIFEST = {
    "schema_version": "pera.policy.v1",
    "name": "my-policy",
    "observation": {
        "cameras": ({"role": "primary", "resolution": (256, 256), "required": True},),
        "language": True,
    },
    "control_frequency_hz": {"trained": 20.0, "accepts": (10.0, 30.0)},
    "action": {
        "kind": "chunk", "horizon": 8,
        "translation": {"format": "delta", "frame": "base"},
        "rotation": {"format": "euler_xyz"},
        "gripper": {"format": "continuous_neg1_1", "close": "high"},
    },
}

def act(obs, task):
    # obs["images"]["primary"]: HxWx3 uint8 (upright), obs["state"]: np.ndarray | None
    return my_model(obs, task.instruction)   # -> (horizon, 7) action chunk

out = pera.evaluate(act, "libero-spatial@1", url=PERA_URL, api_key=KEY,
                    policy_manifest=MANIFEST,
                    tasks=[0], episodes_per_task=2)   # smoke-test scale
print(out["episodes"], "->", out["run_id"])

Two episodes cost a minute and catch convention bugs before they cost an afternoon; when the videos look right, widen to the full protocol (tasks=list(range(10)), episodes_per_task=50 for the LIBERO suites).

No GPU is needed on your side until a real model enters the loop — the simulator runs server-side, so you can prove your whole integration from a laptop.

Stateful policy (internal history, action queues, per-episode reset)? Drive the loop yourself with pera.make() — see docs/03-stateful-and-chunked-policies.md.

The manifest is the contract

You declare what your policy consumes and emits — cameras, state format and rotation convention, action space, control rate. The platform resolves your manifest against the benchmark's:

• direct — formats match, nothing touches your data;
• adapted — explicit, recorded transforms (e.g. a resize), listed in the result bundle;
• refused — incompatible, before any simulation runs or credits burn.

A mismatched pair can never silently produce garbage — that's the design goal. The convention bugs that inspired it (quaternion double-cover at θ=π, a wrist camera served in the wrong frame, upside-down training images) are documented in docs/02-manifests-and-contracts.md, along with a checklist for your own integration.

Evidence you can check

Every evaluation exports a bundle: seeds, per-step action transcripts, per-episode outcomes, failure videos, and per-file checksums under a root hash. The bundle format is an open spec. Verify any published bundle:

pera verify path/to/bundle

See docs/04-verifying-bundles.md. The spec itself is published in spec/ — bundle format, wire protocol, format registry, manifest schemas and every launch protocol.

Tutorials — runnable

tutorials/ has three notebooks plus the actual clients behind every published leaderboard row:

• 01 · Quickstart — manifest → evaluation → results
• 02 · Explore & verify bundles — fully offline: dissect a real vendored evidence bundle, tamper with it, watch verification fail, play the episode video
• 03 · Stateful policy loops — the make() pattern for models with history/action queues
• tutorials/*_client.py — the production integrations (OFT, GR00T, Octo, SmolVLA, π0.5, X-VLA, PulseVLA), each documenting the convention that mattered

New here? Start with ONBOARDING.md — the seven-step path from install to a score you can trust.

Reference docs

1. Quickstart — first evaluation, reading results
2. Manifests & contracts — declaring your policy honestly
3. Stateful & chunked policies — make() loops, replanning
4. Verifying bundles — the evidence format
5. Walkthrough: SmolVLA on LIBERO-Spatial — a complete worked example, from pip install to a score you can compare against our published row
6. Walkthrough: π0-FAST on LIBERO-Spatial — a full 500-episode run, same-machine reference anchor, and evidence-led investigation

Status & access

The hosted platform is in early access with waitlisted API keys. The SDK, bundle spec, and these docs are Apache-2.0. The evaluation engine itself is not open source — published scores are verified precisely because pera runs them; the open spec plus pera verify is how you audit us.

pera is a verapulse project.