A robot-use agent uses a robot just as a computer-use agent uses a computer. OpenRUA is the open harness for one: type openrua run panda "pick up the red cube" and Claude Code or Codex opens in a terminal on the robot's ROS 2 graph, lists the topics, reads the docs in its workspace, writes a script with rclpy, runs it, and checks the camera.

OpenRUA supports these workflows:

• Play in simulation. openrua run brings up a Franka Panda in MuJoCo; the agent drives it the same way it would a real one.
• Put an agent on your robot. Draft its file from the robot's live graph, finish the TODO lines, openrua run <name>. See docs/your-own-robot.md.
• Chat from a browser and terminal. openrua serve keeps a robot and agent conversation running while clients share a message queue and browse saved observations. This experimental feature is included from version 0.1.0; follow Your first shared robot session.
• Run experiments. openrua bench plays a benchmark across tasks and seeds with a fresh sandbox per trial and archives every command the agent ran. See docs/running-experiments.md.

OpenRUA turns your robot into a coding project: your agent explores it like a live codebase, pulls sensor streams into files for reading, and runs commands and programs to move it.

Start playing with your robot like you code a project :)

Quick start

Install, choose a simulator and an agent once, name a robot, run:

pip install openrua
openrua config set --sim robosuite --agent claude-code   # your defaults
openrua build --sim robosuite                          # simulator image
openrua build sandbox --distro humble --agent claude-code
openrua build proxy --agent claude-code
openrua run panda "pick up the red cube"

The robot comes up on its ROS 2 graph, the agent opens on its terminal with that sentence, and the robot powers off when you leave; run first prints which robot, scene and agent it picked.

openrua doctor tells you what is missing before the first run (Docker, the images, an agent login); the details are in docs/install.md.

Browser and shared CLI chat (experimental)

An optional chat-first TUI supports streaming replies and an expandable queue: install pip install -U 'openrua[tui]>=0.1.0', then attach to a running service with openrua chat --tui --name shared.

Use run for the agent's original terminal, or serve for a shared conversation that stays active while browser and CLI clients disconnect and reconnect:

# Terminal A; selected agent images and login must already be ready
openrua serve panda --sim robosuite --name shared --agent codex
# Terminal B, on the same execution host
openrua session --name shared web
openrua chat --name shared "Inspect the scene without moving the robot."

Messages share one queue. You can inspect saved images and programs, edit queued instructions, interrupt and review before continuing, and end the session while retaining its files. Start with the step-by-step tutorial for installation, image preparation, SSH access, expected results, and shutdown. The session manual covers every operation. The original agent terminal cannot yet attach to the same shared conversation; remote hosting and a native mobile app are not implemented.

Choosing what to run

• The world. Without --bench the scene is the simulator's own (robosuite: a table and a cube). --bench libero_pro loads a benchmark's world instead, a LIBERO kitchen with a bowl, a plate, a wine bottle, a drawer and a stove; --task-suite and --task-id pick a scene inside it.
• The agent. --agent codex opens Codex; --model picks the model.
• Once or every time. Whatever config set stored can also be given on the command line: openrua run panda --sim robosuite --bench libero_pro "pick up the bowl". openrua robots, openrua simulators, openrua benchmarks and openrua agents list the choices; openrua benchmarks libero_pro lists one benchmark's suites and tasks.
• A robot that stays up. The same three steps as separate commands: openrua up, then openrua agent "..." in a second terminal, then openrua down.

How it works

 your terminal                       the robot (real or simulated)
 ┌─────────────────────────┐        ┌──────────────────────────────┐
 │ openrua run             │        │ ROS 2 graph                  │
 │  └─ Claude Code / Codex │  DDS   │  /joint_states  /tf  /camera │
 │      in a sandbox with  │◄──────►│  FollowJointTrajectory       │
 │      ros2 · rclpy · docs│        │  GripperCommand  MoveIt      │
 └─────────────────────────┘        └──────────────────────────────┘

• The interface is the robot's own. The agent sees the topics, actions and services the robot exposes, plus machine.yaml (joints, limits, frames, ports) and four short docs. It never sees OpenRUA.
• The sandbox is a plain Ubuntu + ROS 2 container with the agent installed, a workspace mounted, and a whitelist proxy as its only way out (the model API; nothing else).
• A robot is a file (openrua/configs/robots/): the facts true of it wherever it runs (joints, limits, frames, gripper, ports). Your real robot is the same kind of file with a machine: section that says how to reach its graph, passed by path.
• A simulator is a file (openrua/configs/simulators/): the engine, its install, its native scene, and how it drives each robot it embodies.
• A benchmark is a file (openrua/configs/benchmarks/): which robot and simulator, which suites and init states to load, how a trial runs and stops. openrua bench runs trials, checks every promise the workspace docs make before the agent starts, and records each trial with full provenance.
• Everything is checked against one schema (openrua config schema): a misspelled key in any file is an error, never a silent no-op. Your defaults live in ~/.openrua/config.yaml.

Supported robots

A robot is what is true of it wherever it runs: joints, limits, frames, gripper, ports, planner. Which simulator embodies it, and what surrounds it, come from the other two kinds of file.

openrua robots prints this list from the files on disk, yours included.

Supported simulators

A simulator file knows the engine and how it drives each robot it embodies; it knows no benchmark. Its install, and the benchmarks' own over it, is what openrua build renders into one image per declaration (openrua-sim-<name>: ROS 2, the checkouts, the assets, the Python environment); docs/simulation.md describes them.

Supported benchmarks

A benchmark names its robot and simulator and brings its own world: install: (its image contents and ROS distro) and scenes: (scene cameras, and robot embodiments its assets add). openrua benchmarks prints this list; openrua bench --config <name> runs one.

Supported agents

Bring your own agent. An agent is a manifest (how to install its CLI in the sandbox, which hosts it talks to, how it logs in) and a small hooks class (how to launch it); everything else is optional. Pass yours as a path (--agent ./my-agent.yaml) or send a pull request; openrua agents lists what is available and what each can do. See docs/agents.md.

Results

Detailed results will be released with the paper.

Use your own robot

Draft a profile from the robot's live graph, finish the TODO lines, and point up at it:

openrua probe --host > my-ur5.yaml   # joints, limits, frames, ports, cameras from the graph
openrua run ./my-ur5.yaml "..."      # or: openrua config set --robot ./my-ur5.yaml

The profile's machine: section is what the agent's machine.yaml is generated from: model, joint names and limits, frames, gripper, and the ports (trajectory, gripper, twist, wrench) the robot serves. Details in docs/your-own-robot.md.

Simulation and benchmarks

The simulated robots run the community benchmark scenes unchanged; their original success predicates score the trial in place.

openrua bench --config libero_pro --run-id demo \
            --task-suite libero_goal_task --task-ids 0,1 --seeds 0 --operator agent

Every trial writes result.json (verdict, preflight, termination, token accounting), provenance.json (code and simulator commits, image digests, config and prompt hashes), the agent's full transcript, and the workspace it left behind; the run directory keeps a SUMMARY.md regenerated from those files after every trial. Building the simulator checkouts: docs/simulation.md.

A trial replays from its own commands.sh, and a replay with --record renders as a video, terminal on the left, cameras on the right (openrua demo <trial>; see running-experiments.md).

Architecture

Eight units, one direction of dependency: robot/ (the machine, simulated or real), sandbox/ (the agent's terminal and workspace), proxy/ (the only route out), agents/ (the agent contract, registry and launcher), runner/ (bring-up, preflight, operator, verdict, record), demo/ (a recorded trial's files rendered into a video), cli/ and doctor/; under all of them the shared leaves config/ (schema, loader, where things live), errors.py and testing.py. Who may import whom is enforced by CI (import-linter and tests/architecture/). The prose is docs/architecture.md.

Documentation

Citation

The paper will be released soon; until then, cite the software:

@software{openrua,
  author  = {{Terminal World Labs}},
  title   = {OpenRUA},
  year    = {2026},
  url     = {https://github.com/terminalworld/OpenRUA},
  license = {Apache-2.0}
}

License

Apache-2.0