OpenRUA connects off-the-shelf coding agents to robots through their native ROS 2 interfaces. The agent works in a workspace containing robot documentation and starter tools, writes perception and control programs, and uses execution feedback to continue the task.

You can chat with the agent through OpenRUA's terminal UI or browser, use its original terminal, or run recorded benchmark experiments. Shared chat lets you send follow-up instructions, inspect tool activity, manage the queue, and revisit saved observations without starting a new robot each turn.

• Start a shared session: OpenRUA TUI, browser, or plain CLI.
• Use the agent's original terminal: launch a task with openrua run.
• Connect your robot: describe its ROS 2 interfaces in a profile.
• Run experiments: fresh trials, benchmark scoring, and recorded artifacts.

Quick start

Chat with a robot

Use a Linux host with Docker or supported Podman setup. The terminal UI is included in the default installation. Automatic session IDs and history selection are included; version 0.5.0 adds workspace browsing and strengthens startup diagnostics:

pip install -U 'openrua>=0.5.0'
openrua

Each ordinary launch creates a new session with an automatic ID; no name is required. The TUI links robot, simulator and benchmark choices using recorded startup checks. Prepare and start saves a valid selection, builds missing images with visible progress, checks the agent login, then starts the session. Existing images are reused; failed builds keep logs for retry. Save & check reports preparation without downloading or starting resources.

The guided environment list offers Panda in the native robosuite Lift scene, Panda in CaP-Bench, and Panda-Omron in RoboCasa365. Their default scenes have passed startup, reset, sensor-stream and no-op control checks. This does not validate every benchmark task or a model completing the task. Other registered profiles remain available through explicit CLI options. See setup validation for evidence and how to add a combination.

OpenRUA starts the existing session service in the background and opens chat. Type an instruction and press Ctrl+S, for example:

Inspect the workspace documentation and describe the scene without moving the robot.

Ctrl+O (or /files) opens saved workspace files without sending an agent instruction. Text files can be previewed in the TUI; the browser displays saved images, and the optional Pi terminal can display them on compatible terminals.

Ctrl+Q leaves the interface while the session keeps running. Use /resume in the startup form or chat to search previous conversations by title or ID. You can also run openrua --resume, or openrua --gui --resume ID to open a running conversation and its saved workspace images in the browser. To edit defaults later, use openrua --setup, openrua config set, or edit the same config.yaml; changes apply to new sessions, not the one already running.

Choose End session when finished. Records and workspace files are retained; run openrua again for a new session. Ended or unavailable conversations open read-only from history; selecting them never restarts robot execution. The full walkthrough gives a concrete simulated RoboCasa365 example with image preparation, queue operations, SSH access, and shutdown.

Shared chat remains experimental. Codex has live shared-session checks; successful Claude Code turns through the shared adapter still need validation. See validation scope.

Run a benchmark and make a demo

For researchers, the CLI runs fresh benchmark trials and saves the code, observations, transcripts, scores, and configuration needed to inspect a run. The following example runs one CaP-Bench Lift trial, records its cameras, and renders a video with terminal commands beside the robot views.

Install the demo dependencies and prepare the environment once (Docker must be running; use your native Claude Code login):

pip install -U 'openrua[demo]>=0.5.0'
CLAUDE_CONFIG_DIR=~/.openrua/credentials/claude-code claude login
openrua build --bench capbench
openrua build sandbox --distro humble --agent claude-code
openrua build proxy --agent claude-code
openrua doctor panda --sim robosuite --bench capbench --agent claude-code

Then run and render:

openrua bench --config capbench --run-id lift-demo \
  --task-suite capbench_lift --task-ids 0 --seeds 0 \
  --operator agent --record --record-every 4 --wall-clock-min 10
openrua demo runs/capbench/lift-demo/trials/capbench_lift-0/seed0 --speed 4

The video is saved as demo.mp4 in that trial directory. Its result.json reports the benchmark verdict, and provenance.json records the resolved configuration and software versions. This small, recorded run demonstrates the method; reproducing the paper's aggregate results requires its full task/seed sets, model settings, and evaluation budgets. Recording also adds rendering cost. The bundled CaP-Bench configuration selects Claude Opus 5 with high reasoning effort; --agent and --model can select another available configuration.

Native login requires the corresponding host CLI. See installation for prerequisites and credentials. To check the benchmark setup without a model call, use the same bench command with --operator none and omit recording; this checks startup and interfaces, not task success. See running experiments for full runs and recorded replays of existing trials, and the paper for the experimental protocol.

Choose your interface

The first three use one shared queue per session. These examples explicitly name a session; omitting --name creates a new one with an automatic ID. To reconnect to a running session, pass only its name and interface choice; startup options cannot change an active conversation. Existing chat and session commands remain available for explicit client operations. Native run starts a separate session using the agent's original TUI; it cannot yet take over that shared conversation. For a native terminal with a robot that stays up between agent visits, use up, agent, and down as described in First task in simulation.

In shared chat, new instructions queue behind the active turn. Interrupting pauses the queue for review; Resume continues it. Ending the session stops its resources while preserving files; deleting them is a separate operation. For development, openrua serve still runs the service in the foreground. Stopping that process ends its session, unlike closing one of its clients.

For an optional keyboard-first Pi terminal, install pip install -U 'openrua[pi]>=0.5.0' and run openrua --tui pi. It provides keyboard configuration, /resume, queue management, and tool expansion on the same shared sessions, with no separate Node install. See the Pi guide for controls and current limits.

How it works

Treat the robot as an interactive software project. Give an off-the-shelf coding agent a task, terminal access to the robot's native ROS 2 interface, and a workspace. The agent explores the machine, writes and tests programs, and uses execution feedback to refine its actions.

flowchart LR
    Task[User task] --> Agent[Off-the-shelf coding agent]
    Agent <--> Workspace[Workspace: docs, starter tools, observations and programs]
    Robot[Robot's native ROS 2 interfaces] -->|Agent pulls sensor data into files| Workspace
    Agent -->|ROS 2 commands and agent-written programs| Robot

• Workspace as harness. Documentation and readable starter tools give the agent a starting point. Files preserve observations, measurements, programs, and notes as the task progresses. OpenRUA prescribes no robot-task workflow.
• Perception as file I/O. The agent pulls sensor data on demand, inspects saved images, and can write code to process images and compute measurements.
• Manipulation as coding. The agent writes and executes commands or programs against native ROS 2 interfaces, incorporating sensor feedback when needed. It decides when to observe, how to act, and how to verify completion.

Task-specific perception and control strategies emerge from the agent's own coding during execution. The paper analyzes these behaviors, including image processing, metric measurement, and feedback controllers, without a learned robot policy or a task-specific primitive library.

For the software modules, plugin boundaries, and shared-session implementation, see Architecture.

Choosing what to run

• Robot and scene. --sim selects the simulator; --bench, --task-suite, and --task-id select a benchmark scene. Without a benchmark selection or saved benchmark default, robosuite uses its native Lift scene.
• Agent and model. Choose --agent and optionally --model when starting the session. Switching them inside a running conversation is not implemented.
• Defaults. Save common choices with openrua config set; explicit command arguments override them. Configuration documents the fields.
• Available extensions. openrua robots, openrua simulators, openrua benchmarks, and openrua agents list the bundled and local choices. openrua benchmarks libero_pro lists its suites and tasks.

Supported robots

A robot is what is true of it wherever it runs: joints, limits, frames, gripper, ports, planner. Which simulator embodies it, and what surrounds it, come from the other two kinds of file.

openrua robots prints this list from the files on disk, yours included.

Supported simulators

A simulator file knows the engine and how it drives each robot it embodies; it knows no benchmark. Its install, and the benchmarks' own over it, is what openrua build renders into one image per declaration (openrua-sim-<name>: ROS 2, the checkouts, the assets, the Python environment); docs/simulation.md describes them.

Supported benchmarks

A benchmark names its robot and simulator and brings its own world: install: (its image contents and ROS distro) and scenes: (scene cameras, and robot embodiments its assets add). openrua benchmarks prints this list; openrua bench --config <name> runs one.

Supported agents

Bring your own agent. An agent is a manifest (how to install its CLI in the sandbox, which hosts it talks to, how it logs in) and a small hooks class (how to launch it); everything else is optional. Pass yours as a path (--agent ./my-agent.yaml) or send a pull request; openrua agents lists what is available and what each can do. See docs/agents.md.

Results

Detailed results and analysis are available in our paper.

Use your own robot

Draft a profile from the robot's live graph, finish the TODO lines, and pass it to run:

openrua probe --host > my-ur5.yaml   # joints, limits, frames, ports, cameras from the graph
openrua run ./my-ur5.yaml "..."      # or: openrua config set --robot ./my-ur5.yaml

The profile's machine: section is what the agent's machine.yaml is generated from: model, joint names and limits, frames, gripper, and the ports (trajectory, gripper, twist, wrench) the robot serves. Details in docs/your-own-robot.md.

Simulation and benchmarks

The simulated robots run the community benchmark scenes unchanged; their original success predicates score the trial in place.

openrua bench --config libero_pro --run-id demo \
            --task-suite libero_goal_task --task-ids 0,1 --seeds 0 --operator agent

Every trial writes result.json (verdict, preflight, termination, token accounting), provenance.json (code and simulator commits, image digests, config and prompt hashes), the agent's full transcript, and the workspace it left behind; the run directory keeps a SUMMARY.md regenerated from those files after every trial. Building the simulator checkouts: docs/simulation.md.

A trial replays from its own commands.sh, and a replay with --record renders as a video, terminal on the left, cameras on the right (openrua demo <trial>; see running-experiments.md).

Architecture

The implementation separates robot resources, native agent adapters, shared conversations, and presentation. runner/ composes resources; sessions/ coordinates messages and events; tui/ and web/ provide client interfaces. Robot, sandbox, proxy, and agent knowledge stay in their respective modules. Benchmark execution and recorded demos have separate entry points.

Import-linter and architecture tests enforce these boundaries. See the module map, agent extension guide, and contribution guide.

Documentation

Citation

If you use OpenRUA in your research, please cite our paper:

@misc{chu2026openrua,
  title         = {{OpenRUA}: Robot-Use Agents Are Zero-Shot Visuomotor Policies},
  author        = {Zhaoyang Chu and Earl T. Barr and Claire Le Goues and Peter O'Hearn and Mark Harman and Federica Sarro and He Ye},
  year          = {2026},
  eprint        = {2610.02459},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url           = {https://arxiv.org/abs/2610.02459}
}

License

Apache-2.0