ros-agent 0.9.0

MCP agent layer for robots — 49 tools, skills, fleet, vision AI for any ROS 2 robot

ros-agent

The MCP agent layer for robots. 49 tools that let Claude, Cursor, or any MCP client control a real ROS 2 robot. Plugin skills for autonomous behaviors. Real-time vision (YOLO + CLIP + JEPA). MuJoCo simulation. Fleet management. Two pip dependencies in the core — no ROS install required on your machine.

Use any AI client you want — Claude Desktop, Claude Code, Cursor, or any MCP-compatible host. Add one line to your config and your AI gets camera feeds, velocity control, sensor data, autonomous patrol, person following, object search, and full ROS 2 introspection. Switch robots by changing an IP address.

MCP-native from the ground up	49 tools, 8 guided prompts, 6 live resources, and a topic template — all exposed through standard MCP. HTTP+SSE and stdio transports. Works with any client that speaks the protocol.
Skills plugin system	5 built-in skills (compose, inspect, follow-me, patrol, scan-detect). Write your own in 10 lines of Python. 4 loading paths: built-in, pip packages, filesystem, project config. Every skill becomes MCP tools automatically.
Real-time vision stack	YOLO object detection + tracking, CLIP zero-shot search ("find the red cup"), JEPA self-supervised attention heatmaps, Cosmos 3 world model (16B, MLX 4-bit on Mac). Toggle models live from the UI.
Any ROS 2 robot, no ROS needed	Connects over rosbridge WebSocket — no ROS installation on the host. Also supports direct DDS via CycloneDDS for zero-latency local transport. Tested with Unitree Go2, G1, TurtleBot, drones, arms.
Built-in Claude + Gemini agent	Streaming tool use with dynamic ROS context injection. Safety velocity clamping. Persistent cross-session memory. Behavioral identity via SOUL.md.
Simulation and fleet	MuJoCo headless physics with trained ONNX locomotion policies. Fleet dashboard for multi-robot management. Blueprint system for robot configurations. CLIP-indexed spatial memory with geo-search.

---

Quick Install

pip install ros-agent
ros-agent

That's it. Browser opens at http://127.0.0.1:8765. Webcam starts with live detection.

From source

git clone https://github.com/publu/RoboRun.git
cd RoboRun
pip install -e .
ros-agent

npx (Node 18+)

npx ros-agent

Optional extras

pip install ros-agent[vision]   # YOLO + CLIP + OpenCV
pip install ros-agent[ros]      # Direct DDS (CycloneDDS)
pip install ros-agent[gemini]   # Gemini agent
pip install ros-agent[all]      # Everything

---

Connect Your AI

Claude Desktop / Cursor (HTTP)

{
  "mcpServers": {
    "ros-agent": {
      "type": "http",
      "url": "http://localhost:8765/mcp"
    }
  }
}

Claude Code / CLI clients (stdio)

{
  "mcpServers": {
    "ros-agent": {
      "command": "ros-agent-mcp"
    }
  }
}

Your AI immediately gets 49 tools for robot control, 8 guided prompts for common workflows, and 6 live resources for introspection.

---

What Your AI Gets

49 Tools

30 built-in ROS tools (topic discovery, pub/sub, service calls, action goals, parameter management, camera snapshots, depth images, velocity commands, node/topic/service/action introspection) plus 19 skill tools from 5 built-in skills.

8 Prompts

Prompt	What it does
explore-robot	Guided discovery of a robot's capabilities
safety-check	Pre-operation safety verification
environment-scan	Full environment survey with camera + sensors
teach-waypoints	Interactive waypoint teaching for patrol
debug-topic	Step-by-step topic diagnosis
quick-start	First-time onboarding in 60 seconds
fleet-sweep	Multi-robot status check
build-workflow	Create a reusable tool chain

6 Resources

Resource	Description
ros-agent://server-info	Server version, uptime, capabilities
ros-agent://skills	Loaded skills and their tools
ros-agent://ros-graph	Live ROS topic/service/node graph
ros-agent://workflows	Saved compose workflows
ros-agent://prompts-catalog	All available prompts
ros-agent://soul	Agent behavioral identity
ros-agent://topic/{path}	Live read from any ROS topic (template)

---

Skills

Plugin-based. Drop a Python file, get new MCP tools.

Built-in                            → ships with ros-agent
ROBORUN_SKILL_PACKAGES=pkg1,pkg2    → pip packages
ROBORUN_SKILL_PATHS=/path/to/dir    → filesystem directories
.roborun/skills.yaml                → project-level config

Built-in Skills

Skill	Tools	What it does
compose	5	Chain tools into reusable workflows — run_sequence, save_workflow, run_workflow
inspect	3	robot_brief (one-call overview), watch_topic (conditional monitor), diff_state (graph changes)
follow_me	1	Visual person-following with P-control on camera feed
patrol	5	Autonomous waypoint patrol loop with configurable dwell times
scan_detect	2	Rotate-and-detect object search using YOLO + CLIP fallback

Write Your Own

SKILL_TOOLS = [
    {"name": "my_tool", "description": "Does a thing",
     "inputSchema": {"type": "object", "properties": {"x": {"type": "string"}}}}
]

def handle(name: str, args: dict) -> str:
    if name == "my_tool":
        return f"Got: {args.get('x')}"

Point ROBORUN_SKILL_PATHS at the directory. Done.

---

Connect a Robot

ros-agent connects to any robot running rosbridge_server over WebSocket. No ROS installation needed on your machine.

# On the robot
ros2 launch rosbridge_server rosbridge_websocket_launch.xml

# From ros-agent — set IP in the UI, or:
curl -X POST http://localhost:8765/api/ros/connect \
  -d '{"host":"192.168.1.100"}'

Direct DDS transport also available for zero-latency local use:

pip install ros-agent[ros]   # adds CycloneDDS + ros_tap

---

Configuration

Variable	Default	Description
ROBORUN_PORT	8765	Server port
ROBOT_IP	—	Robot IP (can also set in UI)
ANTHROPIC_API_KEY	—	Enables Claude agent
GEMINI_API_KEY	—	Enables Gemini agent
ROBORUN_SKILL_PACKAGES	—	Additional skill packages
ROBORUN_SKILL_PATHS	—	Additional skill directories
ROBORUN_MAX_LINEAR_VEL	1.0	Safety velocity limit (m/s)
ROBORUN_MAX_ANGULAR_VEL	1.5	Safety angular limit (rad/s)

---

Contributing

git clone https://github.com/publu/RoboRun.git
cd RoboRun
pip install -e ".[all]"
python -m roborun.server  # internal module name

---

License

MIT — see LICENSE.

Built by Hashing Systems.