rl-mcp

Watch, diagnose and steer a live RL training run — from an AI agent, or from your shell.

Training a policy is a slow feedback loop. Launch, wait an hour, squint at TensorBoard, guess which reward weight was wrong, relaunch. rlmcp shortens it.

Attach it to a running job and you can ask the live process what the robot is doing, look at it, measure how jerky it is, change a reward weight, unlock harder terrain, and roll back if that made things worse. No restart.

Two backends ship: mjlab, which is what this is developed against, and IsaacLab. The simulator sits behind a small adapter, so a third is a couple of hundred lines rather than a fork.

⚠️ Experimental. The surface is real and tested, but it is still moving.

🚀 Start here

If you use a coding agent, give it one line:

Wrap my mjlab env with rlmcp (github.com/mktk1117/rl-mcp) and show me how to use it.

(Say IsaacLab instead if that is what you train on — the docs cover both.)

It reads the docs here, adds the wrapper to your training script, and drives the run from there.

If you are driving yourself, jump to Install and Try it.

What it looks like

rlmcp status                                   # iteration, stage, headline metrics
rlmcp diagnose --seconds 4                     # is the gait smooth? is it tracking?
rlmcp shot --where terrain=pyramid_stairs      # look at a robot on the stairs
rlmcp video --seconds 5                        # and one at 0, 50, 100, 200 … unasked
rlmcp view                                     # where the browser view is (a run has one)
rlmcp set reward.action_rate_l2.weight -0.25 --why "ankles chattering at 15 Hz"
rlmcp run set_terrain terrains='["flat","random_rough"]' max_level=4
rlmcp curriculum advance --why "flat is solved"
rlmcp checkpoint before-experiment             # and `rlmcp load` to undo
rlmcp events --interventions                   # everything anyone did, and why
rlmcp play                                     # watch a finished run's policy

Almost all of that is an MCP tool too, so an agent can drive the whole loop, and it sees the screenshots and plots, not just their file paths. play is the exception, and stays CLI-only because it opens a window.

The CLI knows who is reading. A terminal gets aligned tables, and pictures open by themselves. A pipe gets JSON, and each command's JSON is a parsing contract that does not move — the shapes are written down and pinned by tests, in docs/tools.md.

The whole integration

One call, between building the environment and building the runner.

from mjlab.envs import ManagerBasedRlEnv
import rlmcp

env = ManagerBasedRlEnv(cfg=env_cfg, device="cuda:0", render_mode="rgb_array")
env = rlmcp.wrap(env, session_dir=log_dir / "rlmcp", curriculum="terrain")

vec_env = RslRlVecEnvWrapper(env)
runner  = VelocityOnPolicyRunner(vec_env, agent_cfg, str(log_dir), device)
env.attach_runner(runner)          # PPO knobs, checkpoints, iteration boundaries

runner.learn(num_learning_iterations=agent_cfg.max_iterations)

That is it. Details and traps: docs/your-task.md.

The same one call on IsaacLab

IsaacLab launches its own app before anything else can be imported, so the wrap goes after gym.make. Downstream is the same CLI, the same MCP tools and the same JSON, minus what only mjlab can answer — the terrain commands, which an IsaacLab run reports as absent rather than broken.

env = gym.make(task, cfg=env_cfg, render_mode="rgb_array")
env = rlmcp_isaaclab.wrap(env, session_dir=log_dir / "rlmcp", task_id=task,
                          service_every_steps=agent_cfg.num_steps_per_env)

rlmcp shot --env-id 300, taken mid-training from a stock Isaac-Velocity-Rough-Anymal-D-v0 run: one robot on the stairs, with the other 2047 environments carrying on behind it. Nothing about the task was described to rlmcp — 73 tunable knobs came out of walking the environment's own manager configs, the same walk it uses on mjlab. The example script is examples/train_isaaclab.py, and the things that differ — cameras, the robot's name, which Isaac Sim goes with which driver — are in docs/isaaclab.md.

What you get

A worked example: in-hand cube reorientation

A SharpaWave hand turning a cube to a commanded orientation anywhere in SO(3), then holding it there for a full second. This is the final policy, replayed from its checkpoint with rlmcp play:

Ten runs to get there. Three worked, four were falsified, three were stopped early. That is the normal shape of RL work, which is why rlmcp keeps runs as a record instead of a folder of logs. rlmcp record graph writes one self-contained HTML page — no server, no build step — and it looks like this:

Each card is one run: the clip it produced, and the one line it concluded. Click any of them and the panel on the right shows what that run predicted, what would have proved it wrong, what changed since its parent, and what it measured.

Runs 005 to 007 are the interesting part. The policy froze, because standing still paid about 2600x more than making progress. Cheap drops broke the freeze and produced the mirror failure: the cube thrown off the palm within half a second. Re-pricing the trade in between failed too. Three failures, one conclusion: the reward was never the binding constraint. 008 changed the action interface instead — an EMA filter — and went from 1.1 to 16.8 goals per minute with the cube held 99.4% of the time.

That conclusion is not in any single run. It is in the shape of three siblings, which is the argument for keeping runs as a graph.

The same page has a tree view for the ancestry itself and a parameters view for what changed across runs. More in docs/records.md.

Install

pip install rl-mcp              # training side: numpy, matplotlib, pillow, imageio
pip install 'rl-mcp[server]'    # adds the MCP SDK for the agent side

uv pip install takes the same lines. For what is on main right now, or to skip installing altogether and run the CLI straight from GitHub:

pip install 'rl-mcp[server] @ git+https://github.com/mktk1117/rl-mcp'
uvx --from 'rl-mcp[server] @ git+https://github.com/mktk1117/rl-mcp' rlmcp sessions

A checkout installs as pip install -e '.[server]'.

The MCP SDK is optional on purpose. The training process does not need it, and the server does not need a simulator. Both mcp>=2 and mcp 1.x work.

The simulator is yours to install. rlmcp asks only that it be importable from the training process — for IsaacLab that means installing into the interpreter Isaac Sim runs under.

Register the server with Claude Code:

claude mcp add rlmcp -- rlmcp-server --root /path/to/logs

Try it

rlmcp tasks         # which ids exist here, before anything has run
# train, with the terrain ladder driving itself
rlmcp-train Mjlab-Velocity-Rough-Unitree-G1 --num-envs 4096

Then, from another shell:

rlmcp sessions      # what is running
rlmcp status
rlmcp curriculum
rlmcp diagnose --seconds 4

The explicit-curriculum version of the same run is in examples/train_g1_rough_curriculum.py. On IsaacLab, the equivalent first run is examples/train_isaaclab.py — same four commands afterwards.

Documentation

How it works, briefly

Two processes that never touch each other. They talk through a plain directory of JSON files.

   training process                session directory                 agent process
  torch + a simulator               plain JSON files                  no simulator
┌───────────────────┐             ┌─────────────────┐             ┌──────────────────┐
│ rlmcp-wrapped env │             │ status.json     │             │ MCP server       │
│ publishes metrics │ ──writes──► │ metrics.jsonl   │ ───reads──► │ (38 tools)       │
│ records frames    │             │ events.jsonl    │             │ rlmcp CLI        │
│ runs commands     │ ◄──reads─── │ artifacts/      │ ◄──writes── │ your own scripts │
│ between batches   │             │ inbox/  outbox/ │             │                  │
└───────────────────┘             └─────────────────┘             └──────────────────┘

Telemetry flows left to right. Commands flow right to left. Nobody shares memory, and nothing blocks the training loop.

That buys four things:

• The agent side never imports torch or a simulator, so starting or killing it cannot disturb training.
• Commands run between rollout batches, so an edit can never race the simulator.
• Everything is inspectable with cat when something goes wrong.
• A run leaves a complete, replayable record behind after it exits.

The same split is why a second backend was cheap. Parameter discovery, trace sampling, metrics and the wrapper are written against the shape mjlab and IsaacLab share; each adapter is only what is genuinely its own — how a robot is found in the scene and how a frame is rendered. More in docs/design.md.

Tests

pytest tests -q     # 855 tests, ~10s, no GPU and no simulator required

License

Apache-2.0 — see LICENSE.