PyRoboReplay

Forensic debugging and mission-replay platform for autonomous robots: replay ROS 2 missions, run causal / root-cause analysis on failures, and perform multispectral (RGB + thermal) forensics. Rust core with Python bindings.

What it is

PyRoboReplay ingests recorded robot missions (ROS 2 bags) into a unified timeline and answers why and what-if questions rather than only where and what: which events preceded a failure, what the perception stack missed, and what the robot should have detected. The heavy lifting is a Rust core (cdylib + rlib) built with PyO3, exposed to Python through maturin. A CLI (replay, analyze, compare, list) and a small Python API are the supported entry points today.

Features

Exposed via CLI / Python (verified, tested):

• Mission replay — interactive timeline scrubber with keyboard controls (replay).
• Reality-gap analysis — probabilistic mismatch scoring, severity classification, evidence aggregation, JSON/text output (analyze --detect-gaps).
• Failure detection and diagnosis — Mission.detect_failures(), Mission.analyze_failure() with hypotheses, confidence, and prioritized actions.
• Mission comparison — side-by-side diff of two bags (compare).
• Topic listing — inspect available topics in a bag (list).
• Real ROS 2 CDR decoding — standard sensor_msgs / nav_msgs / geometry_msgs binary layouts are decoded from real message bytes (src/adapters/cdr.rs); pose confidence is derived from the real position-covariance trace, so localization-loss detection fires on real data.

Implemented in the Rust core, not yet wired to CLI or Python (library API only):

• Causal event graphs, multi-factor causality, incident narratives, evidence-quality scoring.
• Persistent world knowledge (entities, locations, facts across missions) and spatial grounding.
• Multi-mission / longitudinal learning.
• Pluggable detection backends (YOLO / SAM / template fallback).
• Retrospective DINO + SAM open-vocabulary object discovery.
• RGB + thermal multispectral fusion and forensic reconstruction.
• Universal temporal fusion and the Nav2 root-cause inference engine (Phases 14-15).

Who it's for / Use cases

• Warehouse and logistics fleets — reconstruct why a run failed, find missed detections and coverage gaps, compare two strategy revisions.
• Safety and compliance review — produce auditable, saved JSON findings for an incident (analyze --detect-gaps --format json --output report.json).
• Perception R&D — compare perception strategies over matched missions and quantify regressions.
• Field autonomy teams — investigate localization loss, planner oscillation, and sensor disagreement from recorded missions.

What it is not

• Not a real-time monitoring or fleet dashboard. It works on recorded missions, not live streams.
• Most advanced modules (Phases 12-15) are not reachable from the CLI or Python. Use the Rust library directly, or track exposure on the roadmap.
• Phase 11 "PyTerrainMap integration" is not real interop. src/.../pyterrain_bridge.rs is self-contained, unit-tested terrain-modeling logic conceptually inspired by PyTerrainMap; there is no Cargo/pip dependency, no shared serialization format, and no parser for PyTerrainMap's output.
• Not published to crates.io — build the Rust crate from source if you need it directly.
• Not a certified or safety-verified system.

Project status

Beta (Development Status 4). Current source version is 2.11.0; the latest PyPI release is 2.10.2. The CLI and Python API are the tested, supported surface. 843 cargo test --lib unit tests pass (re-verified 2026-09-29), plus 7 real-fixture ROS 2 CDR integration tests. cargo fmt --check is clean. cargo clippy --all-targets --all-features -- -D warnings currently reports ~96 errors, so the CI clippy job runs with continue-on-error and does not gate merges. See ROADMAP_HONEST.md for the implemented/partial/missing breakdown.

Installation

pip install pyroboreplay

# or with uv
uv pip install pyroboreplay

# verify
pyroboreplay --version

Requires Python 3.10+. From source (needs a Rust toolchain):

git clone https://github.com/Mullassery/PyRoboReplay.git
cd pyroboreplay
pip install maturin
maturin develop --release      # install the Python package
# or, for the Rust binary/crate only:
cargo build --release

Quick start

# Interactive timeline scrubber
pyroboreplay replay mission.bag

# Reality-gap analysis, saved as JSON
pyroboreplay analyze mission.bag --detect-gaps --detail --format json --output investigation.json

from pyroboreplay import Mission

mission = Mission.from_ros_bag("warehouse.bag")

failures = mission.detect_failures()
print(f"Failures detected: {len(failures)}")

analysis = mission.analyze_failure(timestamp=1234567890.0)
print(f"Root cause: {analysis.get_primary_hypothesis()}")
print(f"Confidence: {analysis.get_diagnostic_confidence():.0%}")

for action in mission.recommend_actions(timestamp=1234567890.0):
    print(f"[{action.get_priority()}] {action.get_description()}")

Interactive replay keys: Space play/pause, n / → next event, p / ← previous event, ↑ / ↓ speed, ? help, q / Esc quit.

MCAP export

Export a mission timeline to the indexed MCAP robotics log format:

pip install "pyroboreplay[mcap]"

from pyroboreplay import Mission, export_mcap

mission = Mission.from_ros_bag("mission.db3")
export_mcap(mission, "mission.mcap")

Events are written to /events/<sensor> and failures to /failures as JSON messages with their original timestamps, so Foxglove/Rerun and other MCAP readers can open the recording. MCAP (MIT) is an optional dependency.

Rerun mission playback

Export a mission timeline and detected failures to a Rerun recording:

pip install "pyroboreplay[rerun]"

from pyroboreplay import Mission, export_mission

mission = Mission.from_ros_bag("mission.db3")
export_mission(mission, "mission.rrd")
# rerun mission.rrd

Events are logged under events/<sensor> on a time timeline; failures are logged under failures with severity/confidence text. Rerun is an optional Apache-2.0 viewer dependency, not a vendored component.

Alpamayo forensic reports

Turn an offline Alpamayo run (.json, .jsonl, .npz, or a mapping) into a trajectory and reasoning report. Run timestamps are matched against failures already detected in the mission:

from pyroboreplay import (
    build_alpamayo_report,
    load_alpamayo_run,
    write_alpamayo_report,
)

run = load_alpamayo_run("alpamayo-run.json")
report = build_alpamayo_report(run, mission=mission, ground_truth=ego_future_xyz)
write_alpamayo_report(report, "alpamayo-report.md")

The report includes minADE/minFDE per run, per-sample reasoning traces, and correlated failures. No torch or model weights are required; generate the artifacts with the external Alpamayo inference stack.

Usage

pyroboreplay compare mission_a.bag mission_b.bag        # side-by-side
pyroboreplay list mission.bag                            # topics in a bag
pyroboreplay analyze mission.bag --detect-gaps --detail  # findings to stdout

The Python package exports Mission, Event, Failure, Hypothesis, RootCauseAnalysis, Action, FleetStatistics, GeoHotspot (see src/pyroboreplay/__init__.py). The multispectral, retrospective-detection, world-knowledge, and prediction features live in the Rust core and are not yet exposed through these bindings.

Documentation

• docs/CLAUDE.md — product vision and architecture
• docs/API.md — Python and Rust API reference
• docs/ARCHITECTURE.md — phase-by-phase architecture
• docs/QUICKSTART.md — getting started
• docs/STATS_DASHBOARD.md — live metrics dashboard (replay --stats-dashboard)
• examples/ — runnable demos (cargo run --example root_cause_analysis_demo)
• ROADMAP_HONEST.md — implemented vs planned, technical debt
• CHANGELOG.md — notable changes

Known issues and deferred work

Blocked / failed:

• Expose phases 12–15; rerun/MCAP export; clippy gate.

Rejected / deferred:

• Live fleet dashboard.

Full shipped/blocked/rejected detail: QUEUE_STATUS.md.

Contributing

Contributions welcome — see CONTRIBUTING.md for setup, conventions, and PR guidelines. Bugs and ideas: GitHub Issues; questions: GitHub Discussions.

Part of the robotics & autonomy family

PyRoboReplay is one of six complementary, composable tools in the Mullassery robotics & autonomy family. Together they form a single stack: perceive → map → simulate → learn → debug → coordinate the fleet.

Each works standalone, but they're complementary by design. Browse all projects →

License

Apache License 2.0 — see LICENSE.