RPX Benchmark Toolkit

Load RPX data, run reference or custom perception models, and compare accuracy, scene-change robustness and compute cost with Python 3.10–3.12:

python -m pip install 'rpx-benchmark[hub,schemas]'
git clone https://github.com/IRVLUTD/RPX.git
cd RPX
python benchmark/examples/run_depth.py

The example is a CPU-only installation check using synthetic data. Reference checkpoints need their own runtime, weights and GPU resources. The package installation alone does not install every model's dependencies.

For toolkit development, replace the PyPI install with an editable source install from the repository root:

python -m pip install -e './benchmark[hub,schemas]'

Reference models

Use isolated environments because model dependencies conflict. The linked READMEs contain build, checkpoint, cache and execution instructions. Run a model's small inference gate before a full benchmark, and retain its dataset revision and runtime metadata with the results.

For segmentation, detection, open-vocabulary detection, generic grounding, sparse depth and keypoint matching, the toolkit supplies the callable APIs, loaders and evaluators below. There is no bundled full pretrained-model sweep for those tasks. Mask annotation models produce dataset labels; they are not a substitute for a benchmark adapter.

Example, inside the selected model environment, from benchmark/:

python scripts/run_depth.py --model da-v2-large --split easy \
  --repo IRVLUTD/RPX --device cuda --max-samples 10 \
  --output-dir ../rpx_results/da-v2-large/easy-smoke
python scripts/run_video_depth.py --help
python scripts/run_relative_pose.py --help
python scripts/run_tracking.py --help

python benchmark/scripts/run_depth_smoke_matrix.py --list-models lists the canonical depth roster without loading weights. Docker matrix files and adapter registries are the source of model identifiers and pinned checkpoint revisions. Model availability and successful inference are different checks: the offline suite tests adapters with controlled inputs, while GPU gates run real weights.

Bring your own model

Wrap a callable and pass it to a public task runner. The callable may invoke a local model or your inference service; convert its response to the documented NumPy output. Authentication and service configuration belong in your wrapper, not in committed source.

import rpx_benchmark as rpx
from my_model import predict_depth  # your implementation: RGB -> depth in metres

model = rpx.make_numpy_depth_model(predict_depth, name="my-depth")
result, report, paths = rpx.run_monocular_depth(
    rpx.MonocularDepthRunConfig(
        model=model,
        split="easy",
        repo_id="IRVLUTD/RPX",
        revision=rpx.DEFAULT_REVISION,
        device="cpu",  # configure your callable's actual device yourself
        output_dir="rpx_results/my-depth/easy",
    )
)
print(result.aggregated)
print(paths["json"])

For a local dataset, supply manifest_path="/data/manifests/monocular_depth/easy.json" to skip downloading. Manifests name a task and dataset root and list samples with modality paths, scene, phase and difficulty. Local examples and all ten public workflows are exercised in tests/test_user_workflows.py. For relative depth, set depth_output_kind="relative" on the factory; never label arbitrary relative values as metric depth.

Factory docstrings specify shapes and units. For custom preprocessing, batching or framework-specific outputs, implement BenchmarkModel or compose BenchmarkableModel with input/output adapters. The generic tracking API is a frame-based contract; use the production tracking runner for the full clip-based paper protocol and its initialization/scoring rules.

Custom VQA model or API

VQA uses JSONL question manifests rather than the core task manifests. A custom callable receives only ordered image paths and the public prompt; ground-truth answers and boxes are kept inside the evaluator.

from rpx_benchmark.vqa import evaluate_vqa
from my_model import predict_vqa

# predict_vqa(image_paths, prompt, max_new_tokens, output_kind) -> raw text
# image_paths: (target,) or (reference_crop, target), in that order.
report = evaluate_vqa(
    "vqa_manifest.jsonl", predict_vqa,
    model_name="my-vlm", model_revision="my-checkpoint-or-service-version",
    image_cache=".cache/rpx-vqa", output_dir="rpx_results/my-vlm",
)
print(report["metrics"])

Return one JSON object with label and bbox (XYXY coordinates normalized to 0–1000); binary questions require yes or no. Parsing failures and inference errors remain in the denominator. Runs save raw outputs, errors, timing, manifest hash, model revision and scored results. Use a fresh output directory. The VQA runtime README explains manifest preparation and the reference-model benchmark. The preserved Molmo adapter in scripts/vqa_models/molmo_backend.py is optional and outside the frozen roster.

Data and results

The Hub loader fetches only required modalities and reuses its cache. Pin a dataset commit for comparable runs; do not mix results from moving revisions. The data README identifies the local canonical splits and metadata. Dataset publishing tools are documented separately in rpx_benchmark/dataset_hub/README.md.

Generic frame runs write result.json, summary.md, cells.parquet and per_sample_metrics.parquet. Video runs write clip/cell results. Hardware, precision and batch size belong in each cell; unavailable measurements stay unavailable. Custom API wall time includes transport and is not a GPU-only latency measurement. Production runners add prediction caches, protocol metrics and provenance records.

Testing

From benchmark/, in a clean virtual environment:

python -m pip install -e '.[dev,hub,schemas,hf-datasets,analysis,ar-pose-audit]' \
  imageio imageio-ffmpeg av transformers build
python -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
python -m pip install --no-deps 'git+https://github.com/JonathonLuiten/TrackEval.git@12c8791b303e0a0b50f753af204249e622d0281a'
pytest tests -q --cov=rpx_benchmark --cov-report=term-missing
pytest ../data/mask_annotation/visual_grounding_gt/vqa_gt/tests -q
ruff check rpx_benchmark tests scripts
mypy rpx_benchmark
python -m build

Tests cover task contracts, local datasets, public custom-model workflows, metrics, reports, download behavior, model-call signatures and VQA parsing. They do not download model checkpoints. Run the linked CUDA inference gates on the target machine to validate real models, available memory and dataset access. Sensor capture and annotation UI checks require their own hardware.

Extend the toolkit

See the metric registration guide and task extension guide. The guides include runnable examples, validation rules, and the source changes needed for a new task identity. They are also bundled in the installed package under rpx_benchmark/guides/.

Browse the toolkit documentation for installation, model integration, and API reference pages.

To locate the bundled guides after installation:

from importlib.resources import files
print(files("rpx_benchmark").joinpath("guides/tasks/README.md").read_text())

To build the documentation from a repository checkout:

pip install -e "./benchmark[hub,schemas,docs]"
python tools/docs/build_pages.py --output _site
python -m http.server --directory _site 8000

Open http://localhost:8000/toolkit-docs/. The build generates guides and API pages from the same source.