WorldBench

Regression testing for video-based robotics world models.

WorldBench compares baseline and candidate checkpoints on the same prediction suite, then surfaces the individual episodes and future horizons that became worse.

Did this new world-model checkpoint improve, and what became worse?

Same fixed episodes
        ->
Baseline predictions vs candidate predictions
        ->
Per-episode and per-horizon deltas
        ->
CI-ready PASS or FAIL

Real Checkpoint Regression Proof

WorldBench's strongest committed proof compares two NanoWM checkpoints from the same model family on the same fixed RT-1 / Fractal episode suite.

WorldBench detected that the candidate improved in aggregate while still surfacing the episode that regressed: episode_002.mp4 changed by -0.33.

This is a fixed 10-episode validation proof, not a public cross-model ranking or a claim of universal model quality.

Artifacts and documentation: artifacts/checkpoint_validation/, docs/checkpoint_validation.md, and docs/checkpoint_regression.md.

Evaluate Two Saved Videos in One Command

If you only have one ground-truth video and one generated prediction video, start here:

python -m pip install "worldbench[video]"

worldbench eval-videos \
  --ground-truth ground_truth.mp4 \
  --prediction predicted_future.mp4 \
  --output results/

This is a beginner-friendly way to evaluate one saved predicted robot future against its matching ground-truth future. It is not, by itself, checkpoint regression testing. To test checkpoint regression, evaluate baseline and candidate predictions separately against the same ground truth, then use eval-batch and gate.

The command decodes both videos, aligns future frames by index, resizes the prediction to the ground-truth resolution when needed, rejects unsafe frame-count mismatches, runs every metric that is valid for video-only inputs, and writes:

results/
  result.json
  summary.md
  artifacts/comparison.png

Expected terminal output resembles:

WorldBench Saved Video Evaluation
Frames: ground truth 8 | prediction 8 | scored 8
Composite Score: 96.10/100
Metric coverage: 2 of 5 configured metrics

Metric Scores
Visual Similarity   94.7
Temporal Stability  97.9
Action Consistency  N/A
Object Permanence   N/A
Contact Realism     N/A

Saved JSON: results/result.json
Saved Markdown: results/summary.md
Saved comparison image: results/artifacts/comparison.png

The Composite Score is a weighted summary of available metrics for this specific ground-truth/prediction pair. It is not accuracy, not task success, and not a universal robotics capability score. For video-only inputs, Visual Similarity and Temporal Stability are usually available. Action Consistency needs action semantics, Object Permanence needs reliable object tracking metadata, and Contact Realism needs reliable robot/object contact tracking; without those signals they are reported as N/A and excluded from the weighted score.

Try a no-file demo:

worldbench eval-videos --demo --output results/demo

Colab notebook: examples/colab/worldbench_saved_video_demo.ipynb

Details: docs/SAVED_VIDEO_EVALUATION.md

Quickstart

From a repository checkout, install the package and run the committed sample dataset:

python -m pip install -e ".[video]"
worldbench validate examples/demo_dataset
worldbench eval examples/demo_dataset --predictions examples/demo_dataset/good_model --output-root .worldbench/readme-quickstart
worldbench report artifacts/real_model_eval/nanowm_rt1_episode0.json --output .worldbench/readme-quickstart/nanowm_report.md

The sample dataset is synthetic and exists to verify the local setup. The committed real-model NanoWM artifact is documented below.

Who It Is For

WorldBench is for researchers and engineers who train video-based robot world models and release multiple prediction checkpoints. It is useful when a team already has a fixed validation rollout suite, is relying on ad hoc comparison scripts, and needs reproducible regression reports before accepting a new checkpoint.

For the primary checkpoint-regression workflow, users provide:

• matching ground-truth future videos
• baseline checkpoint predictions
• candidate checkpoint predictions
• consistent episode alignment

A single prediction set can still be evaluated with eval-video or eval-batch, but the main product workflow is baseline-versus-candidate regression testing.

Checkpoint Regression Workflow

1. Freeze a fixed suite of robot episodes.
2. Generate predictions with the baseline checkpoint.
3. Generate predictions with the candidate checkpoint.
4. Run WorldBench on matching episodes.
5. Inspect aggregate, episode, and horizon changes.
6. Use the gate result to accept or reject the candidate.

worldbench eval-batch \
  --ground-truth eval_suite \
  --predictions baseline_predictions \
  --name baseline \
  --skip-context 4 \
  --output baseline.json

worldbench eval-batch \
  --ground-truth eval_suite \
  --predictions candidate_predictions \
  --name candidate \
  --skip-context 4 \
  --output candidate.json

worldbench gate \
  --baseline baseline.json \
  --candidate candidate.json \
  --strict-config-match \
  --max-episode-regressions 0

eval-video scores one aligned ground-truth/prediction video pair. eval-batch scores one checkpoint folder across matching episode videos. gate compares baseline and candidate batch artifacts and returns PASS or FAIL.

Results include JSON output, metric coverage, effective normalized weights, unavailable metric reasons, and per-horizon metric summaries when enough frames are available. compare, report, and dashboard provide local comparison, Markdown, and HTML inspection surfaces.

Scores, Regressions, And Benchmarks

An isolated WorldBench Composite Score does not establish that one unrelated model is universally better than another. The reliable use is:

same dataset
same episodes
same preprocessing
same context
same prediction horizon
baseline versus candidate

Cross-model or cross-dataset score comparisons may be invalid when protocols differ.

General robotics benchmarks ask: how capable is this model or robot?

WorldBench asks: did this new video-based world-model checkpoint regress?

These workflows are complementary. Standardized benchmarks compare models under a shared public protocol; WorldBench tests a team's own model changes on its own fixed prediction suite.

Compatibility

Directly compatible:

• models that export aligned predicted future RGB frames or videos for robot episodes
• action-conditioned robot video predictors
• image-to-video robot world models
• visual dynamics models
• latent world models with an RGB decoder
• simulators or learned models that render predicted visual futures

Requires an adapter before WorldBench can score it correctly:

• robot-specific action vectors
• state-trajectory predictions
• latent-only outputs
• native 3D, 4D, or point-cloud predictions

Not the current target:

• action-only policies
• VLAs that do not predict future observations
• text-only environment models
• symbolic planners
• closed-loop robot-task evaluation

WorldBench evaluates saved predicted visual futures. It does not run robot control, execute real-world tasks, or provide a universal measure of robot capability.

Reproducible NanoWM RT-1 Integration

WorldBench includes a lightweight adapter and guide for evaluating public NanoWM RT-1 checkpoint predictions after NanoWM has generated RGB rollout clips. A one-episode pilot reproduced the 50k-versus-300k comparison on a 16 GB CPU-only Mac with no paid compute; the 10-episode validation above remains the flagship proof.

See: docs/NANOWM_RT1.md

Real-Model Integration Proof

A separate single-rollout integration proof verifies that actual NanoWM RT-1 generated future frames can pass through WorldBench. It is retained as integration evidence, not as the primary product proof.

This is a single-rollout integration proof, not a public cross-model ranking and not a claim that NanoWM is 92.4% accurate.

Artifact: artifacts/real_model_eval/nanowm_rt1_episode0.json

Supported Metrics And N/A Behavior

WorldBench does not invent semantic scores. A metric returns N/A when the rollout does not provide the semantics or signals needed for reliable evaluation. Unsupported metrics are excluded from the weighted overall score, and the remaining available weights are renormalized. Visual Similarity alone does not prove task success or physical correctness.

Default weights are Visual Similarity 0.25, Action Consistency 0.30, Temporal Stability 0.20, Object Permanence 0.15, and Contact Realism 0.10.

The NanoWM artifact had Visual Similarity and Temporal Stability available. Its rounded score comes from:

(89.24097498284189 * 0.25 + 96.32682361785054 * 0.20) / 0.45 = 92.39024104284573

Details: docs/metric_support.md

Real-Data Validation

The repository records native LeRobot validation against chocolat-nya/yaskawa-untangle-dataset episode 0 in network-marked integration tests. Those tests are not part of the default offline pytest run.

Source evidence: tests/test_lerobot_integration.py and docs/real_data_validation.md.

LeRobot Support

import-lerobot supports native Hugging Face LeRobot datasets through the optional lerobot extra and a legacy local LeRobot-style folder converter.

worldbench import-lerobot --repo-id chocolat-nya/yaskawa-untangle-dataset --episodes 0:1 --camera observation.images.fixed_cam1 --timeline video --out examples/yaskawa_video
worldbench import-lerobot --repo-id chocolat-nya/yaskawa-untangle-dataset --episodes 0:1 --camera observation.images.fixed_cam1 --timeline control --out examples/yaskawa_control

--timeline video exports one timestep per unique source camera frame. It aligns actions with latest_at_or_before_timestamp and states with nearest_timestamp.

--timeline control exports one timestep per source control row. It aligns actions and states with source_control_row.

Imported actions, states, and metadata preserve source control indices, source control timestamps, source video frame indices, source video timestamps, repo id, camera key, timeline, and alignment strategy when available.

Details: docs/real_data_validation.md

Corruption Validation

Committed corruption artifacts were generated from the recorded Yaskawa video-timeline dataset. Scores decreased monotonically across the tested corruption severities. Temporal scrambling produced a smaller effect than frame freezing in these artifacts.

Frame-freeze artifact: artifacts/frame_freeze_benchmark.json

Temporal-scramble artifact: artifacts/temporal_scramble_benchmark.json

CLI Reference Summary

Run worldbench COMMAND --help for the installed version's exact options.

Python API Summary

from worldbench import WorldBench

result = WorldBench("examples/demo_dataset").evaluate(
    predictions="examples/demo_dataset/good_model"
)
print(result.score)
result.save_json(".worldbench/readme-quickstart/result.json")

Public SDK exports include WorldBench, WorldModelRun, Metrics, evaluate, load_dataset, EvaluationResult, and MetricResult.

Dataset Format

The frame-dataset layout is:

dataset/
  episode_001/
    frames/
    predictions/
    actions.json
    states.json
    metadata.json

actions.json records timestep t, optional timestamps/provenance, an action value, optional dx/dy, and optional gripper state. states.json records timestep t, optional timestamps/provenance, optional observation_state, and optional synthetic tracker coordinates. metadata.json stores episode identity, robot/task labels, FPS, and optional LeRobot provenance fields.

The video workflow accepts one video per episode. Ground truth and prediction folders are paired by identical relative POSIX paths, with matching future-frame count after --skip-context, resolution, and FPS.

Details: docs/DATA_FORMAT.md

Current Limitations

• The public single-rollout NanoWM artifact covers one rollout and eight generated future frames.
• The NanoWM artifact is not a public cross-model ranking or model accuracy claim.
• Arbitrary numeric robot actions require explicit semantics or an action adapter before Action Consistency is meaningful.
• Real-world Object Permanence and Contact Realism require reliable tracking support.
• Temporal scrambling currently causes a smaller score decrease than frame freezing in the committed corruption artifacts.
• WorldBench evaluates saved predictions; it does not run arbitrary model inference.
• WorldBench performs offline prediction evaluation, not closed-loop task execution.
• Normal CI does not download LeRobot datasets or run network integration tests.

Roadmap

Working now

• direct video-pair evaluation
• multi-episode batch evaluation
• per-horizon metric summaries
• regression gate
• native LeRobot import
• single-rollout NanoWM integration artifact
• frame-freeze and temporal-scramble corruption artifacts

Next

1. explicit action-adapter registry
2. evaluate more real-model rollouts
3. evaluate a second real model
4. get an external user

Later

• adapters for simulator-rendered prediction videos
• adapters for common robot-video export formats
• shared reports

Details: docs/ROADMAP.md

Contributing

Use a supported Python version, install the development dependencies, and run the local checks before sending changes:

python -m pip install -e ".[dev]"
ruff check .
pytest
python -m build

Network LeRobot tests are marked integration and are excluded from the default test command.

License

WorldBench is licensed under Apache-2.0. See LICENSE.