rpx-benchmark 0.3.7

Same Scene, Different Story:
Evaluating Robot Perception Across Scene Phases in the Wild

Score a perception model on 100 real scenes recorded before, during and after manipulation,
and see where it breaks with phase robustness Φ and worst-phase quality 𝒥_min.

Getting started

The rpx-benchmark package provides the data, an adapter per task, the paper's benchmark protocol, and the Φ and 𝒥_min calculator. You write one function that runs your model.

pip install 'rpx-benchmark[hub,schemas]'

The example below scores an image-depth model on a 6-frame trial. The download fetches only this task's files (about 160 MB) and needs no login.

import numpy as np
import rpx_benchmark as rpx

# 1. Download the task's data
manifest = rpx.download_split("monocular_depth", "easy", max_samples=6)

# 2. Your model: RGB image (H, W, 3) in, depth (H, W) in metres out
def predict(rgb):
    return np.ones(rgb.shape[:2], np.float32)

# 3. Wrap it in the task's adapter and run the benchmark
model = rpx.make_numpy_depth_model(predict, name="my-model")
result, report, paths = rpx.run_monocular_depth(rpx.MonocularDepthRunConfig(
    model=model, manifest_path=str(manifest), output_dir="rpx_results/my-model",
))
print(result.aggregated["absrel"], result.aggregated["delta1"])

Φ and 𝒥_min compare phases, so they need all three phases of many scenes. Drop max_samples to download the full tier, run the benchmark again, then:

from rpx_benchmark.examples.summarize_metrics import calculate, read_records

metrics = ["absrel", "rmse", "silog", "delta1"]
summary = calculate(read_records(paths["json"]), metrics)

To try any task before writing code, rpx.benchmark("T2", rpx.download("T2", max_samples=2)) runs it with a placeholder model.

Tasks

Every task follows the steps above with its own download name, adapter and runner.

_{* Pass manifest_name="relative_pose" as well. Functions are under rpx; vqa is rpx_benchmark.vqa, and its models return text, so they need no adapter. dummy_models.py shows the inputs and outputs of every task.}

python -m rpx_benchmark.examples.benchmark_tasks \
    --task T1 --split easy --max-samples 6 --model my_model:predict

# Check the install without downloading anything
python -m rpx_benchmark.examples.benchmark_tasks --task all --smoke

To add a task or a metric, see the toolkit docs on tasks and metrics.

Model contracts and paper models

The table lists what your function receives and returns for each task, and the Docker image that contains every model the paper evaluated on it.

dummy_models.py has a runnable placeholder for each row. Models that predict relative depth pass depth_output_kind="relative" to the adapter. The model guide documents the full input contracts, and the benchmark pages walk through each task with figures.

For T5 and T6, rpx.download("T5") returns a 58-question quick start and rpx.download("T5", full=True) the paper's full list (21,000 questions; 10,500 for T6). Pass either, or your own list, to evaluate_vqa; the next section covers models behind an API.

Always pull a specific tag: the bare irvlutd/rpx image is an unrelated legacy image. The Docker guide also covers the box- and text-prompted trackers and GemDepth, which ships as a build recipe only.

The generic T3 and T4 runners report per-sample errors. The paper's HOTA and pooled pose AUC come from the clip- and pair-level reference runners in benchmark/scripts.

Custom VQA model or API

Any local model or hosted API works as long as it returns text:

from rpx_benchmark.vqa import evaluate_vqa

def predict_vqa(image_paths, prompt, max_new_tokens, output_kind):
    # image_paths is (target,) or (reference_crop, target).
    # Return your model's raw text.
    ...

report = evaluate_vqa(
    "vqa_manifest.jsonl", predict_vqa,
    model_name="my-vlm", model_revision="v1",
    image_cache=".cache/rpx-vqa", output_dir="rpx_results/my-vlm",
)
print(report["metrics"]["bbox_accuracy_at_0_5"])

The reply should be one JSON object with label and bbox (XYXY, normalized to 0–1000). The model never sees the answers, and replies that fail to parse count as misses. The VQA guide explains how to prepare the manifest.

Results

In the paper, quality drops once hands enter the scene, and none of the 65 evaluations is both accurate and robust across phases.

Further reading

• Φ and JEDI calculator: compute Φ and 𝒥_min from existing results
• Hardware profiler: latency, throughput and memory
• Mask annotation pipeline: label your own scenes the way RPX was labelled
• Metrics and tasks: add your own
• Dataset card and data guide: 100 scenes, 70 objects, 133,121 frames

Testing

python -m pip install -e '.[dev,hub,schemas,hf-datasets,analysis,ar-pose-audit]' \
    imageio imageio-ffmpeg av transformers build
python -m pip install torch torchvision \
    --index-url https://download.pytorch.org/whl/cpu
TRACKEVAL=12c8791b303e0a0b50f753af204249e622d0281a
python -m pip install --no-deps \
    "git+https://github.com/JonathonLuiten/TrackEval.git@$TRACKEVAL"

pytest tests -q
ruff check rpx_benchmark tests scripts
mypy rpx_benchmark
python -m build

Citation

@misc{rpx2026,
  title  = {Same Scene, Different Story: Evaluating Robot Perception Across
            Scene Phases in the Wild},
  author = {{Jishnu Jaykumar P} and Kadosh, Itay and Vijayakumar, Narendhiran and
            Kamath, Srinanditha and Allu, Sai Haneesh and Rangappa, Govind Tyagi and
            Maheshwari, Animesh and Wang, Jikai and Xiang, Yu},
  year   = {2026},
  note   = {Dataset: \url{https://huggingface.co/datasets/IRVLUTD/RPX}}
}

The toolkit is MIT-licensed and the dataset is CC BY 4.0. Pretrained models keep their own licenses.