UMI Whole-System Runtime + Tools

Current system description:

• docs/UMI_SYSTEM_DESCRIPTION.md
• docs/UMI_USER_STORIES.md
• docs/UMI_TASK_BREAKDOWN.md
• docs/UMI_SYSTEM_ARCHITECTURE.md
• docs/UMI_STATE_MACHINE.md

Read in this order if you want the current whole-system definition:

1. docs/UMI_SYSTEM_DESCRIPTION.md
2. docs/UMI_USER_STORIES.md
3. docs/UMI_TASK_BREAKDOWN.md
4. docs/UMI_SYSTEM_ARCHITECTURE.md
5. docs/UMI_STATE_MACHINE.md
6. docs/UMI_SYSTEM_SOLUTION_SKELETON.md

This repo provides:

• A Rerun-based live visualizer (apps/rerun_visualizer.py)
• A Rerun-based three-point session review tool for head / left palm / right palm playback in browser (apps/rerun_three_pose_review.py)
• A PyPI-packaged Rerun review CLI (umi-rerun-review) and remote session-selection service (umi-rerun-session-review) that open Rerun Web Viewer directly from RDK-hosted data
• The current PICO pose path plus legacy Quest compatibility readers (vr_umi_sdk/pico/, vr_umi_sdk/quest/)
• A device-side recording and export runtime (vr_umi_sdk/) + CLI (apps/record.py)
• A board-local debug helper + optional debug web shell for setup and field debugging
• Supporting docs for the evolving RDK X5 whole-system design

For convenience, the old root-level scripts (vr_umi_ui.py, vr_umi_record.py, etc.) are kept as thin wrappers.

PyPI Rerun review package

The PyPI package is intentionally self-contained for field review. After:

python3 -m pip install Delta-UMI-Viewer==0.4

the installed environment provides:

umi-rerun-review --session /path/to/session
umi-rerun-session-review --host 0.0.0.0 --port 8768 --session-root 运行数据=/home/sunrise/vr_umi_runs

For systemd deployments, point ExecStart= at the installed umi-rerun-session-review binary. No repo clone or PYTHONPATH is required for the Rerun picker service.

Integrated target runtime profile

The integrated target whole-system lane in this repo is now expressed through:

• board profile: config/board_profiles/pico_dual_runtime.json
• startup contract: config/startup_self_check_pico_dual_runtime.json

That target shape is:

• 1 Pico
• 2 servo ports
• 2 cameras (4 camera nodes expected by startup check)
• 2 buttons
• 2 lights
• board power voltage present on the runtime power socket
• robot-gateway WebSocket updates over apps/umi_robot_gateway.py

The systemd deployment units under deploy/systemd/ now point at this integrated profile/config pair.

For wave-1 packaging prep, treat the shipped systemd lane as the authoritative packaged topology:

• umi-runtime.service
• umi-button.service
• umi-board-power.service
• umi-board-light.service
• umi-debug-helper.service

Code-level CLI defaults that still point at config/board_profiles/current_validation_board.json and config/startup_self_check.json remain available for local development / compatibility only. They are not the packaged defaults; deploy/systemd/ overrides them to the integrated pair above.

Separate export-worker packaging is still deferred in this checkout. Runtime/export sidecars exist, but there is no shipped deploy/systemd/umi-export-worker.service unit in the current packaged lane.

The next gate after wave-1 packaging prep is on-target validation:

• docs/ON_TARGET_ACCEPTANCE_CHECKLIST.md
• docs/ON_TARGET_ACCEPTANCE_RUNBOOK.md
• docs/ON_TARGET_ACCEPTANCE_EVIDENCE_TEMPLATE.md

Live visualization

1. Start the visualizer (captures local cameras + listens for pose JSON):

python vr_umi_rerun_visualizer.py --listen 0.0.0.0:9001

Legacy Quest note (the active project target is now PICO; keep these steps only for the old Quest visualizer path):

adb devices
adb forward tcp:5050 tcp:5050

2. Start the Quest TCP reader and forward poses to the visualizer:

python tcp_quest_reader.py --visualizer 127.0.0.1:9001 --send-hz 60 --auto-send

Notes:

• If you omit --auto-send, use L/R Trigger to start sending; other buttons stop.
• Camera devices:
  • Linux: /dev/video0, /dev/video2, ...
  • macOS: 0, 1, ...

Recording a session (dataset)

python vr_umi_record.py --cam-fps 15 --sample-hz 60 --action-mode vr_pose

Optional (USB grippers):

python vr_umi_record.py --gripper-left-port /dev/ttyUSB0 --gripper-right-port /dev/ttyUSB1 --gripper-baudrate 115200

This writes a session folder under data/vr_umi_sessions/<timestamp>/. See docs/DATASET_FORMAT.md.

Non-Pico bring-up lane

For board-side bring-up without Pico, use the dedicated non-Pico entrypoint:

python -m apps.non_pico_bringup \
  --board-config config/board_profiles/non_pico_dual_bringup.json \
  --acceptance-stage stage_gate \
  --duration-s 60 \
  --cam-fps 30 \
  --gripper-left-port /dev/ttyUSB0 \
  --gripper-right-port /dev/ttyUSB1 \
  --print-summary

To inspect what the board currently auto-detects before a real run:

python -m apps.non_pico_bringup --print-discovery

Notes:

• This lane is for 2 buttons + 2 lights + 2 servo reads + 2 cameras.
• It keeps Pico out of scope and does not change startup self-check.
• The bundled dual-button / dual-light board profile is a provisional bring-up profile aligned to the current schematic shape: two 3-pin button connectors and two 4-pin light connectors. Keep treating the runtime-side naming as provisional until the real on-device mapping is fully confirmed.
• The run writes the normal sidecars plus bringup_summary.json, button_events.jsonl, and light_events.jsonl.

Legacy Local UI Shell

python vr_umi_ui.py

Notes:

• This is a legacy local development shell kept for compatibility.
• It is not the target whole-system recording interface.
• Formal whole-system control should go through the device runtime / CLI path.
• The target UI role is setup, debugging, status inspection, and guided repair.

Convert to LeRobotDataset (v3)

python vr_umi_to_lerobot.py --input data/vr_umi_sessions --output data/lerobot/vr_umi_dataset --overwrite

One-command viz pipeline

python apps/viz_pipeline.py --session data/vr_umi_sessions/<session_name>

Repo layout

• vr_umi_sdk/: runtime, hardware, capture, export, and debug helper code
• apps/: CLI entrypoints and board-local/local debug helper shells
• docs/: system description, architecture, state machine, requirements, and dataset docs
• tools/: development utilities (camera smoke/tuner)
• examples/: small scripts