PhysicalAI MuJoCo SO-101 Plugin
MuJoCo SO-101 simulation plugin for PhysicalAI, the Python library and runtime for robot control, transport, and CLI workflows. It registers with Physical AI Studio, the application that discovers catalog plugins and provides robot setup, teleoperation, and workflow experiences. This plugin lets you run a virtual SO-101 robot as a PhysicalAI transport owner, then connect to it from Studio exactly like real hardware. Part of the physicalai-plugins monorepo.
Features
- Run a virtual SO-101 as a PhysicalAI transport owner
- MuJoCo interactive viewer
- Camera streams over HTTP (MJPEG) and optional v4l2loopback
- REST control server (scenes, reset, shutdown)
- Built-in task scenes
Screenshots
Placeholder images — replace them with real screenshots.
What this is for
- Teleoperating a virtual SO-101 from PhysicalAI Studio.
- Testing robot setup, task logic, and control flows without physical hardware.
- Playing back policy/inference outputs in a simulated scene while monitoring robot + camera streams.
- Developing workflows where Studio, transport, and robot APIs stay identical between sim and real deployments.
Quick start
From the repo root:
uv run --no-sync physicalai-mujoco-so101 start
By default this starts:
- A MuJoCo SO-101 owner named
mujoco-so101 - Viewer window enabled
- Camera streams served over HTTP (MJPEG + REST control server on port
8080) - Control rate
50 Hz - Substeps
10
Then open PhysicalAI Studio and connect to the robot type MuJoCo SO-101 Follower with name mujoco-so101.
CLI teleoperation (self-relay)
With the owner running, you can also relay the simulation back to itself using the PhysicalAI CLI:
uv run physicalai run --config packages/physicalai-mujoco-so101-plugin/examples/runtime/teleop.yaml
Press Ctrl+C to stop.
View the camera streams in a browser or with curl:
curl http://127.0.0.1:8080/health
MJPEG stream URLs are http://127.0.0.1:8080/cameras/<name>/mjpeg (e.g. open
http://127.0.0.1:8080/cameras/overview/mjpeg in a browser, or play it in VLC).
CLI options
uv run --no-sync physicalai-mujoco-so101 start --help
Common options:
--name <robot-name>: transport name (must match Studio payload)--model <path>: custom XML/URDF path (bypasses scene resolution)--scene <name>: scene name (single_pick_place,pick_lift,pick_place, oryahtzee, defaultsingle_pick_place)--no-gui: disable MuJoCo interactive viewer--no-cameras: disable camera rendering entirely (HTTP streams and v4l2loopback)--http-host <host>: host for the camera/control HTTP server (default127.0.0.1)--http-port <port>: port for the camera/control HTTP server (default8080)--no-http: disable the camera/control HTTP server--v4l2: also publish cameras to v4l2loopback devices (requiresmodprobe v4l2loopback)--rate-hz <float>: owner loop frequency--substeps <int>: MuJoCo steps per control cycle--idle-timeout <seconds>: seconds with zero subscribers before self-exit (default10without HTTP, disabled when HTTP is enabled so stream viewers keep the sim alive)--allow-remote: allow non-loopback zenoh connections
Cameras over HTTP (default)
The plugin renders two camera feeds and serves them over HTTP:
wrist->http://127.0.0.1:8080/cameras/wrist/mjpegoverview->http://127.0.0.1:8080/cameras/overview/mjpeg
Each camera is also available as a single JPEG snapshot at
http://127.0.0.1:8080/cameras/<name>/frame.jpg.
REST control API
The HTTP server exposes control endpoints for resetting and switching scenes:
# List available scenes and the current one
curl http://127.0.0.1:8080/scenes
# Switch to another scene
curl -X POST http://127.0.0.1:8080/scenes/pick_place
# Reset/randomize the current scene
curl -X POST http://127.0.0.1:8080/reset
# Stop the simulation owner
curl -X POST http://127.0.0.1:8080/shutdown
| Endpoint | Method | Description |
|---|---|---|
/ |
GET | Service info, endpoint index |
/health |
GET | Sim status: connected, current scene, cameras |
/cameras |
GET | Camera list with stream/snapshot URLs |
/cameras/{name}/mjpeg |
GET | MJPEG stream (multipart/x-mixed-replace) |
/cameras/{name}/frame.jpg |
GET | Latest frame as a JPEG snapshot |
/scenes |
GET | Current scene and available scene IDs |
/scenes/{scene_id} |
POST | Switch to another registered scene |
/reset |
POST | Reset/randomize the current scene |
/shutdown |
POST | Gracefully stop the simulation owner |
Because the owner process is detached, stopping the CLI with Ctrl+C does not
stop the simulation. Use POST /shutdown, --idle-timeout, or kill the owner
process directly.
Cameras and v4l2loopback (opt-in)
For workflows that need a webcam-visible device, pass --v4l2 to publish the
same camera feeds to v4l2loopback devices:
wrist->/dev/video<wrist-video-id>(default/dev/video60)overview->/dev/video<overview-video-id>(default/dev/video62)
Example with custom IDs:
uv run --no-sync physicalai-mujoco-so101 start --v4l2 --wrist-video-id 70 --overview-video-id 71
One-time setup (Linux)
Install and load v4l2loopback:
sudo modprobe v4l2loopback exclusive_caps=1 video_nr=60,61
If you use custom camera IDs, use matching video_nr values. Example:
sudo modprobe v4l2loopback exclusive_caps=1 video_nr=70,71
If the module is already loaded with different params, unload and reload:
sudo rmmod v4l2loopback
sudo modprobe v4l2loopback exclusive_caps=1 video_nr=60,61
Verify devices
ls /dev/video60 /dev/video61
For custom IDs, verify those device nodes instead.
Optional sanity checks:
v4l2-ctl --all -d /dev/video60
v4l2-ctl --all -d /dev/video61
Troubleshooting
HTTP server unavailable or port already in use
- The camera/control server binds to
--http-host/--http-port(default127.0.0.1:8080). - If the port is taken the simulation continues without HTTP and logs a warning — pass a different
--http-port. - When running multiple simulations, give each a distinct
--http-port.
Camera '<name>' unavailable or invalid argument on /dev/video* (with --v4l2)
- Ensure v4l2loopback is loaded with
exclusive_caps=1 - Ensure the configured video devices exist (default
/dev/video60,/dev/video62) - Check permissions on device nodes
Viewer opens but has Wayland warnings (libdecor, window position)
These warnings are typically non-fatal on Wayland and can be ignored if simulation continues.
Camera/control server is not started
- Confirm the sim is running and the port is free:
curl http://127.0.0.1:8080/health --no-httpor--http-port 0disables the server;--no-camerasdisables all camera rendering
Using with Studio teleop and inference playback
Typical workflow:
- Start simulation owner:
physicalai-mujoco-so101 start - Connect from PhysicalAI Studio (
MuJoCo SO-101 Follower) - Teleoperate in Studio and observe state/cameras
- Run policy inference and play action outputs into the same simulated robot
Because this uses PhysicalAI transport + Studio catalog integration, you can iterate on control and inference loops in simulation before moving to hardware.
Scenes
The plugin ships with built-in scenes that provide different environments for the robot:
| Scene ID | Description | Free objects | Target |
|---|---|---|---|
single_pick_place (default) |
One block and a target disc | 1 cube | target disc |
pick_lift |
Three colored cubes and a target disc | 3 cubes | target disc |
pick_place |
A cube, a cylinder, and a target zone | cube + cylinder | target zone |
yahtzee |
Six dice and a cup | 6 dice | cup |
Start with a specific scene:
uv run --no-sync physicalai-mujoco-so101 start --scene pick_place
Keyboard shortcut: cycle scenes
When the MuJoCo viewer is open, press n (next scene) to cycle through available scenes. The scene switches at the next control cycle:
- Loads the new scene XML
- Updates the existing viewer with the new environment
- Resets block joints, target bodies, and spawn parameters
--model bypasses scene resolution entirely; only the exported scene XML path is loaded, and scene cycling is unavailable.
Current status and future improvements
Planned/desired improvements:
- More configurable camera presets (pose/FOV/fps via CLI or payload)
- Scene randomization presets (object layouts, target marker variants, textures)
- RTSP streaming sink (the frame-buffer plumbing is sink-agnostic; an RTSP
server such as
aiortspor MediaMTX can consume the same buffers later) - Optional richer sensor streams (depth/segmentation style outputs)
- Additional sample tasks and policy playback recipes in this package
Metadata
Release files for physicalai-mujoco-so101-plugin 0.3.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| physicalai_mujoco_so101_plugin-0.3.1.tar.gz | 10.7 MB | With provenance | Details for physicalai_mujoco_so101_plugin-0.3.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| physicalai_mujoco_so101_plugin-0.3.1-py3-none-any.whl | Python 3 | none | any | Details for physicalai_mujoco_so101_plugin-0.3.1-py3-none-any.whl |
Total release size: 21.5 MB
Release files / physicalai_mujoco_so101_plugin-0.3.1.tar.gz
| Download URL | physicalai_mujoco_so101_plugin-0.3.1.tar.gz |
|---|---|
| Size | 10.7 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ad34cdb841ee61994b2694424424bc8dfcd93a0790f2bbbb94b514c5bef35b31
|
|
BLAKE2b-256 checksum How to use checksums |
911aaf864f1b1e34daebeffdd6e1d6bcebf46c6c1184f3e57981316240dfa648
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.
Transparency logRelease files / physicalai_mujoco_so101_plugin-0.3.1-py3-none-any.whl
| Download URL | physicalai_mujoco_so101_plugin-0.3.1-py3-none-any.whl |
|---|---|
| Size | 10.8 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1c684f701621410773d8f2a99f13f887e8fa5b2662252898ebf98412a45d69fd
|
|
BLAKE2b-256 checksum How to use checksums |
8e7835d1c62701b6b64c65fa8c941a0f63ab3de89e30020a24e8ddf4c7246675
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.
Transparency logSecurity for physicalai-mujoco-so101-plugin 0.3.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for physicalai_mujoco_so101_plugin-0.3.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.