A robot-use agent (RUA) uses a robot just as a computer-use agent uses a computer.

OpenRUA connects off-the-shelf coding agents to robots through their native ROS 2 interfaces. The agent works in a workspace containing robot documentation and starter tools, writes perception and control programs, and uses execution feedback to continue the task.

You can chat with the agent through OpenRUA's terminal UI or browser, use its original terminal, have it do one task on its own, recorded, or run recorded benchmark experiments. Shared chat lets you send follow-up instructions, inspect tool activity, manage the queue, and revisit saved observations without starting a new robot each turn.

• Start a shared session: OpenRUA TUI, browser, or plain CLI.
• Run one task, or use the agent's original terminal: openrua exec records the agent doing a task on its own; openrua agent opens its own terminal on the robot.
• Connect a real robot: openrua connect, your coding agent setting it up with you.
• Describe your robot: its ROS 2 interfaces in a profile.
• Run experiments: fresh trials, benchmark scoring, and recorded artifacts.

OpenRUA turns your robot into a coding project: your agent explores it like a live codebase, pulls sensor streams into files for reading, and runs commands and programs to move it.

Start playing with your robot like you code a project :)

🔥 News

• [2026-10-23] 📂 Run a task in the same robot project. openrua exec "<task>" reuses your code and notes, with a separate record for each run. bench keeps isolated evaluation workspaces. (v0.20.0)
• [2026-10-22] 📁 Your robot is a coding project. Open OpenRUA in a directory, keep programs and notes there, and continue in a new conversation the next day. Resume searches this project's history; --all shows every project. (v0.19.0)
• [2026-10-22] 🧰 Bring OpenRUA to your own robot, starting from a bare computer. Missing Docker or an agent login? openrua sets them up with you, and your coding agent connects the robot step by step. (v0.18.2)
• [2026-10-22] 👀 Watch your robot through its own eyes. The browser shows its cameras live while the agent works, and every real-robot openrua exec run is recorded, ready to become a demo video with openrua demo. (v0.17.0)
• [2026-10-21] 🎯 Hand your agent a task and let it work on its own. openrua exec runs it in simulation or on a real robot and keeps a full record of the run. (v0.14.0)
• [2026-10-21] 🦾 Your coding agent can now set up a real robot for you. Run openrua connect: Claude Code or Codex works out what the robot is, builds its ROS 2 drivers and writes its config with you, and OpenRUA checks every step. (v0.13.0)
• [2026-10-21] ⌨️ Talk to your robot the way you talk to Claude Code and Codex. Send a follow-up while the agent works, switch /permissions and /model on the fly, and press Esc to stop the agent and the arm's motion together. (v0.13.0)
• [2026-10-21] 🤖 Real robots, ready to go. Each robot describes itself in its own config, and OpenRUA builds its drivers and its own sandbox the first time you start it. (v0.11.0, v0.12.0)
• [2026-10-06] 💬 A keyboard terminal for your robot becomes the default way to chat with it. (v0.6.0)
• [2026-10-03] 👥 Shared robot conversations: one session you can join from the terminal, the browser or a script. (v0.1.0)
• [2026-10-01] 📄 The paper is out: OpenRUA: Robot-Use Agents Are Zero-Shot Visuomotor Policies.
• [2026-09-14] 🚀 OpenRUA is live. pip install openrua and give Claude Code or Codex a robot, simulated or real. (v0.0.7)

Quick start

Before you start, have on a Linux computer:

• Docker: the robot, the simulator and your agent's terminal run in containers.
• Claude Code or Codex, signed in with your account: your coding agent does the work.

Missing either one? openrua sets it up with you.

pip install -U openrua
mkdir my-robot-project
cd my-robot-project
openrua

Your project directory. OpenRUA asks whether you trust this workspace; you can use it, choose another directory or create one. Programs, observations and notes stay here across sessions. Opening the same directory again starts a new conversation; --resume finds this project's earlier ones, and --resume --all searches every project (robot projects).

A simulated robot. In the setup menu, pick a robot and a scene, then Prepare and start. The first start builds the images, which takes a few minutes. Then tell the agent what you want, for example:

Look through the cameras and describe the scene, without moving the robot.

A real robot. Choose Connect a real robot (or run openrua connect). Your agent sets the robot up with you, step by step, and OpenRUA checks each step, up to a first small motion with the emergency stop at hand (Connect a real robot).

While it works. Enter sends, Esc stops the agent and the arm's motion, /watch opens the robot's cameras live in the browser and /help lists the rest (terminal guide). Ctrl+D asks whether to keep the session running; openrua --resume brings you back.

Next

• Keep useful methods. Ask the agent to save a project skill and reuse it in a later conversation. openrua skills lists saved methods and controls how they accumulate. openrua skills review looks back across completed project work and saves skill proposals for review (skills guide). In a project conversation, /project opens saved skills, review reports and explicit adoption. /model shows model choices; Codex /model effort adjusts reasoning effort for the next turn.
• One task, on its own. openrua exec "<task>" works in the same project and records the run, as claude -p or codex exec do (first task).
• Benchmarks and demo videos. Run a benchmark's tasks, scored by its own success predicates, and render a trial as a video (running experiments).
• Your robot, described by hand. Draft its profile from the live graph instead of connecting it with your agent (your own robot).

Choose your interface

The first three share one conversation and its queue: join from any of them, leave, and come back with openrua --resume (shared sessions). The terminal guide has the keys and commands. To work in the agent's own original terminal instead, run openrua agent (or openrua agent codex): it starts a robot when none is up (first task).

How it works

Treat the robot as an interactive software project. Give an off-the-shelf coding agent a task, terminal access to the robot's native ROS 2 interface, and a workspace. The agent explores the machine, writes and tests programs, and uses execution feedback to refine its actions.

your terminal                                 the robot (real or simulated)
┌──────────────────────────────────┐          ┌────────────────────────────────┐
│ openrua                          │          │ ROS 2 graph                    │
│  └─ Claude Code / Codex          │   DDS    │ /joint_states · /tf · /camera  │
│      in a sandbox workspace      │◄────────►│ FollowJointTrajectory          │
│      ros2 · rclpy · docs         │          │ GripperCommand · MoveIt        │
│      starter tools · saved files │          │ Velocity control               │
└──────────────────────────────────┘          └────────────────────────────────┘

• Workspace as harness. Documentation and readable starter tools give the agent a starting point. Files preserve observations, measurements, programs, and notes as the task progresses. OpenRUA prescribes no robot-task workflow.
• Perception as file I/O. The agent pulls sensor data on demand, inspects saved images, and can write code to process images and compute measurements.
• Manipulation as coding. The agent writes and executes commands or programs against native ROS 2 interfaces, incorporating sensor feedback when needed. It decides when to observe, how to act, and how to verify completion.

Task-specific perception and control strategies emerge from the agent's own coding during execution. The paper analyzes these behaviors, including image processing, metric measurement, and feedback controllers, without a learned robot policy or a task-specific primitive library.

For the software modules, plugin boundaries, and shared-session implementation, see Architecture.

Choosing what to run

• Robot and scene. --sim selects the simulator; --bench, --task-suite, and --task-id select a benchmark scene. Without a benchmark selection or saved benchmark default, robosuite uses its native Lift scene.
• Agent and model. Choose --agent and optionally --model when starting the session. /model <name> changes the model from the next request; the agent cannot be switched inside a running conversation.
• What the agent can reach. A session you start sets no limits: the agent reaches the internet as on your machine. A recorded run (openrua exec, openrua bench) limits it to its model service by default. --internet, --[no-]web-search and --[no-]account-connectors set either (what the agent can reach).
• Custom profiles. Pass a file with --robot, --sim, or --bench to use your own environment in the TUI. It is validated and prepared without being replaced by a guided choice. See your own robot.
• Defaults. Save common choices with openrua config set; explicit command arguments override them. Configuration documents the fields.
• Available extensions. openrua robots, openrua simulators, openrua benchmarks, and openrua agents list the bundled choices; openrua robots also lists the robots connected on this computer. openrua benchmarks libero_pro lists its suites and tasks.

Supported robots

A robot is what is true of it wherever it runs: joints, limits, frames, gripper, ports, planner. Which simulator embodies it, and what surrounds it, come from the other two kinds of file.

openrua robots prints this list, and the robots connected on this computer.

Supported simulators

A simulator file knows the engine and how it drives each robot it embodies; it knows no benchmark. Its install, and the benchmarks' own over it, is what openrua build renders into one image per declaration (openrua-sim-<name>: ROS 2, the checkouts, the assets, the Python environment); docs/simulation.md describes them.

Supported benchmarks

A benchmark names its robot and simulator and brings its own world: install: (its image contents and ROS distro) and scenes: (scene cameras, and robot embodiments its assets add). openrua benchmarks prints this list; openrua bench --config <name> runs one.

Supported agents

Experimental external plugins are available for Kimi Code and ZCode. Both have passed small real-provider file tasks and two-turn shared-chat checks. Their API paths have real-provider checks; native Kimi OAuth and ZCode Coding Plan key profile staging have offline checks only. ZCode OAuth reuse remains unsupported, and robot tasks remain unvalidated. They are not default setup choices.

Bring your own agent. An agent is a manifest (how to install its CLI in the sandbox, which hosts it talks to, how it logs in) and a small hooks class (how to launch it, and how to hold a conversation, which OpenRUA's terminal, browser and openrua connect need). Pass yours as a path (--agent ./my-agent.yaml) or send a pull request; openrua agents lists what is available and what each can do. See docs/agents.md.

Results

Results below are from our paper, evaluated zero-shot in simulation without privileged simulator state or access to the benchmark's success predicate during execution.

Main benchmark results use Claude Code with Claude Opus 5 at high reasoning effort:

The model comparison uses the LIBERO-10 suite of LIBERO-PRO, with both perturbation types and 200 trials per configuration:

See the paper for baselines, evaluation protocols and detailed analysis.

Use your own robot

The quickest way is openrua connect my-arm: your coding agent finds out what the robot is, builds and starts its ROS 2 drivers and writes robot.yaml with you, and OpenRUA checks each step. When every check has passed, the same conversation goes on in the robot's own terminal. See docs/connect.md.

To write the profile yourself, draft it from the robot's live graph, finish the TODO lines, and pass it where a robot is named:

openrua probe --host > my-ur5.yaml   # joints, limits, frames, ports, cameras from the graph
openrua --robot ./my-ur5.yaml        # chat with the agent on it; or: openrua config set --robot ./my-ur5.yaml
openrua exec ./my-ur5.yaml "put the cup on the plate"   # one task on its own, recorded

The profile says what the machine is and how its drivers start and stop (your own robot).

Simulation and benchmarks

The simulated robots run the community benchmark scenes unchanged; their original success predicates score the trial in place. openrua exec runs one trial; openrua bench runs a benchmark's task set, one trial per task and seed:

openrua bench --config libero_pro --run-id demo \
            --task-suite libero_goal_task --task-ids 0,1 --seeds 0 --operator agent

Every benchmark trial keeps its verdict, provenance, the agent's transcript and the workspace it left. Project exec runs retain their workspace in the project itself. A trial replays from its own commands; a replay renders as a video, terminal on the left and cameras on the right (running experiments).

Architecture

The implementation separates robot resources, native agent adapters, shared conversations, and presentation. runner/ composes resources; sessions/ coordinates messages and events; terminal/, ui/terminal/ and web/ provide client interfaces. Robot, sandbox, proxy, and agent knowledge stay in their respective modules. Benchmark execution and recorded demos have separate entry points.

Import-linter and architecture tests enforce these boundaries. See the module map, agent extension guide, and contribution guide.

Documentation

Citation

If you use OpenRUA in your research, please cite our paper:

@misc{chu2026openrua,
  title         = {{OpenRUA}: Robot-Use Agents Are Zero-Shot Visuomotor Policies},
  author        = {Zhaoyang Chu and Earl T. Barr and Claire Le Goues and Peter O'Hearn and Mark Harman and Federica Sarro and He Ye},
  year          = {2026},
  eprint        = {2610.02459},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url           = {https://arxiv.org/abs/2610.02459}
}

License

Apache-2.0

Feedback

Try OpenRUA with your coding agent and robot, and tell us how it goes. Bug reports, confusing steps, and ideas for improving the experience are welcome in GitHub Issues. If something fails, include your OpenRUA version, agent, environment, and the command or steps that led to it. Please remove credentials and private data from any logs you share.