Skip to main content

PhysicalAI MuJoCo SO-101 Plugin

MuJoCo SO-101 simulation plugin for PhysicalAI, the Python library and runtime for robot control, transport, and CLI workflows. It registers with Physical AI Studio, the application that discovers catalog plugins and provides robot setup, teleoperation, and workflow experiences. This plugin lets you run a virtual SO-101 robot as a PhysicalAI transport owner, then connect to it from Studio exactly like real hardware. Part of the physicalai-plugins monorepo.

Features

  • Run a virtual SO-101 as a PhysicalAI transport owner
  • MuJoCo interactive viewer
  • Camera streams over HTTP (MJPEG) and optional v4l2loopback
  • REST control server (scenes, reset, shutdown)
  • Built-in task scenes

Screenshots

Placeholder images — replace them with real screenshots.

MuJoCo SO-101 in the PhysicalAI Studio robot catalog

Connecting to the MuJoCo SO-101 in PhysicalAI Studio

MuJoCo viewer with a pick-and-place scene

MJPEG camera stream in a browser

What this is for

  • Teleoperating a virtual SO-101 from PhysicalAI Studio.
  • Testing robot setup, task logic, and control flows without physical hardware.
  • Playing back policy/inference outputs in a simulated scene while monitoring robot + camera streams.
  • Developing workflows where Studio, transport, and robot APIs stay identical between sim and real deployments.

Quick start

From the repo root:

uv run --no-sync physicalai-mujoco-so101 start

By default this starts:

  • A MuJoCo SO-101 owner named mujoco-so101
  • Viewer window enabled
  • Camera streams served over HTTP (MJPEG + REST control server on port 8080)
  • Control rate 50 Hz
  • Substeps 10

Then open PhysicalAI Studio and connect to the robot type MuJoCo SO-101 Follower with name mujoco-so101.

CLI teleoperation (self-relay)

With the owner running, you can also relay the simulation back to itself using the PhysicalAI CLI:

uv run physicalai run --config packages/physicalai-mujoco-so101-plugin/examples/runtime/teleop.yaml

Press Ctrl+C to stop.

View the camera streams in a browser or with curl:

curl http://127.0.0.1:8080/health

MJPEG stream URLs are http://127.0.0.1:8080/cameras/<name>/mjpeg (e.g. open http://127.0.0.1:8080/cameras/overview/mjpeg in a browser, or play it in VLC).

CLI options

uv run --no-sync physicalai-mujoco-so101 start --help

Common options:

  • --name <robot-name>: transport name (must match Studio payload)
  • --model <path>: custom XML/URDF path (bypasses scene resolution)
  • --scene <name>: scene name (single_pick_place, pick_lift, pick_place, or yahtzee, default single_pick_place)
  • --no-gui: disable MuJoCo interactive viewer
  • --no-cameras: disable camera rendering entirely (HTTP streams and v4l2loopback)
  • --http-host <host>: host for the camera/control HTTP server (default 127.0.0.1)
  • --http-port <port>: port for the camera/control HTTP server (default 8080)
  • --no-http: disable the camera/control HTTP server
  • --v4l2: also publish cameras to v4l2loopback devices (requires modprobe v4l2loopback)
  • --rate-hz <float>: owner loop frequency
  • --substeps <int>: MuJoCo steps per control cycle
  • --idle-timeout <seconds>: seconds with zero subscribers before self-exit (default 10 without HTTP, disabled when HTTP is enabled so stream viewers keep the sim alive)
  • --allow-remote: allow non-loopback zenoh connections

Cameras over HTTP (default)

The plugin renders two camera feeds and serves them over HTTP:

  • wrist -> http://127.0.0.1:8080/cameras/wrist/mjpeg
  • overview -> http://127.0.0.1:8080/cameras/overview/mjpeg

Each camera is also available as a single JPEG snapshot at http://127.0.0.1:8080/cameras/<name>/frame.jpg.

REST control API

The HTTP server exposes control endpoints for resetting and switching scenes:

# List available scenes and the current one
curl http://127.0.0.1:8080/scenes

# Switch to another scene
curl -X POST http://127.0.0.1:8080/scenes/pick_place

# Reset/randomize the current scene
curl -X POST http://127.0.0.1:8080/reset

# Stop the simulation owner
curl -X POST http://127.0.0.1:8080/shutdown
Endpoint Method Description
/ GET Service info, endpoint index
/health GET Sim status: connected, current scene, cameras
/cameras GET Camera list with stream/snapshot URLs
/cameras/{name}/mjpeg GET MJPEG stream (multipart/x-mixed-replace)
/cameras/{name}/frame.jpg GET Latest frame as a JPEG snapshot
/scenes GET Current scene and available scene IDs
/scenes/{scene_id} POST Switch to another registered scene
/reset POST Reset/randomize the current scene
/shutdown POST Gracefully stop the simulation owner

Because the owner process is detached, stopping the CLI with Ctrl+C does not stop the simulation. Use POST /shutdown, --idle-timeout, or kill the owner process directly.

Cameras and v4l2loopback (opt-in)

For workflows that need a webcam-visible device, pass --v4l2 to publish the same camera feeds to v4l2loopback devices:

  • wrist -> /dev/video<wrist-video-id> (default /dev/video60)
  • overview -> /dev/video<overview-video-id> (default /dev/video62)

Example with custom IDs:

uv run --no-sync physicalai-mujoco-so101 start --v4l2 --wrist-video-id 70 --overview-video-id 71

One-time setup (Linux)

Install and load v4l2loopback:

sudo modprobe v4l2loopback exclusive_caps=1 video_nr=60,61

If you use custom camera IDs, use matching video_nr values. Example:

sudo modprobe v4l2loopback exclusive_caps=1 video_nr=70,71

If the module is already loaded with different params, unload and reload:

sudo rmmod v4l2loopback
sudo modprobe v4l2loopback exclusive_caps=1 video_nr=60,61

Verify devices

ls /dev/video60 /dev/video61

For custom IDs, verify those device nodes instead.

Optional sanity checks:

v4l2-ctl --all -d /dev/video60
v4l2-ctl --all -d /dev/video61

Troubleshooting

HTTP server unavailable or port already in use

  • The camera/control server binds to --http-host/--http-port (default 127.0.0.1:8080).
  • If the port is taken the simulation continues without HTTP and logs a warning — pass a different --http-port.
  • When running multiple simulations, give each a distinct --http-port.

Camera '<name>' unavailable or invalid argument on /dev/video* (with --v4l2)

  • Ensure v4l2loopback is loaded with exclusive_caps=1
  • Ensure the configured video devices exist (default /dev/video60, /dev/video62)
  • Check permissions on device nodes

Viewer opens but has Wayland warnings (libdecor, window position)

These warnings are typically non-fatal on Wayland and can be ignored if simulation continues.

Camera/control server is not started

  • Confirm the sim is running and the port is free: curl http://127.0.0.1:8080/health
  • --no-http or --http-port 0 disables the server; --no-cameras disables all camera rendering

Using with Studio teleop and inference playback

Typical workflow:

  1. Start simulation owner: physicalai-mujoco-so101 start
  2. Connect from PhysicalAI Studio (MuJoCo SO-101 Follower)
  3. Teleoperate in Studio and observe state/cameras
  4. Run policy inference and play action outputs into the same simulated robot

Because this uses PhysicalAI transport + Studio catalog integration, you can iterate on control and inference loops in simulation before moving to hardware.

Scenes

The plugin ships with built-in scenes that provide different environments for the robot:

Scene ID Description Free objects Target
single_pick_place (default) One block and a target disc 1 cube target disc
pick_lift Three colored cubes and a target disc 3 cubes target disc
pick_place A cube, a cylinder, and a target zone cube + cylinder target zone
yahtzee Six dice and a cup 6 dice cup

Start with a specific scene:

uv run --no-sync physicalai-mujoco-so101 start --scene pick_place

Keyboard shortcut: cycle scenes

When the MuJoCo viewer is open, press n (next scene) to cycle through available scenes. The scene switches at the next control cycle:

  • Loads the new scene XML
  • Updates the existing viewer with the new environment
  • Resets block joints, target bodies, and spawn parameters

--model bypasses scene resolution entirely; only the exported scene XML path is loaded, and scene cycling is unavailable.

Current status and future improvements

Planned/desired improvements:

  • More configurable camera presets (pose/FOV/fps via CLI or payload)
  • Scene randomization presets (object layouts, target marker variants, textures)
  • RTSP streaming sink (the frame-buffer plumbing is sink-agnostic; an RTSP server such as aiortsp or MediaMTX can consume the same buffers later)
  • Optional richer sensor streams (depth/segmentation style outputs)
  • Additional sample tasks and policy playback recipes in this package

Metadata

Release files for physicalai-mujoco-so101-plugin 0.3.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for physicalai-mujoco-so101-plugin 0.3.1
File Size Uploaded Provenance Actions
physicalai_mujoco_so101_plugin-0.3.1.tar.gz 10.7 MB With provenance Details for physicalai_mujoco_so101_plugin-0.3.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for physicalai-mujoco-so101-plugin 0.3.1
File Interpreter ABI Platform Actions
physicalai_mujoco_so101_plugin-0.3.1-py3-none-any.whl Python 3 none any Details for physicalai_mujoco_so101_plugin-0.3.1-py3-none-any.whl

Total release size: 21.5 MB

Release files / physicalai_mujoco_so101_plugin-0.3.1.tar.gz

Download URL physicalai_mujoco_so101_plugin-0.3.1.tar.gz
Size 10.7 MB
Tags Source
SHA-256 checksum
How to use checksums
ad34cdb841ee61994b2694424424bc8dfcd93a0790f2bbbb94b514c5bef35b31
BLAKE2b-256 checksum
How to use checksums
911aaf864f1b1e34daebeffdd6e1d6bcebf46c6c1184f3e57981316240dfa648
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.

Transparency log

Release files / physicalai_mujoco_so101_plugin-0.3.1-py3-none-any.whl

Download URL physicalai_mujoco_so101_plugin-0.3.1-py3-none-any.whl
Size 10.8 MB
Tags Python 3
SHA-256 checksum
How to use checksums
1c684f701621410773d8f2a99f13f887e8fa5b2662252898ebf98412a45d69fd
BLAKE2b-256 checksum
How to use checksums
8e7835d1c62701b6b64c65fa8c941a0f63ab3de89e30020a24e8ddf4c7246675
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.

Transparency log

Security for physicalai-mujoco-so101-plugin 0.3.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for physicalai_mujoco_so101_plugin-0.3.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page