inspect-robots-agent 0.12.0

LLM agent policy for Inspect Robots: frontier LLMs (Claude, GPT, anything OpenAI-compatible) drive any registered embodiment through tool calls.

inspect-robots-agent

LLM agent policy for Inspect Robots: frontier LLMs (Claude, GPT, anything behind an OpenAI-compatible API) drive any registered embodiment through tool calls, as a first-class Policy named agent. The same policy runs ad-hoc instructions and scores on registered tasks next to fine-tuned VLAs.

Install

pip install inspect-robots inspect-robots-agent

Quickstart (no hardware)

export ANTHROPIC_API_KEY=sk-ant-...

inspect-robots "pick up the cube" --policy agent \
    -P model=anthropic/claude-fable-5 -P effort=low --embodiment cubepick

Model strings are OpenRouter-style provider/model, resolved from -P model=... or $INSPECT_ROBOTS_MODEL. API keys come from the environment:

1. -P base_url=... (with -P api_key_env=NAME): any OpenAI-compatible endpoint
2. A known provider prefix with that provider's key set: the provider's own endpoint, prefix stripped from the model id
3. OPENROUTER_API_KEY: OpenRouter, any model string. Ids ending in a known OpenRouter variant suffix (:free, :nitro, :floor, :extended, :online, :thinking) always route here, since the variant means nothing to a provider's own API; other colons (fine-tune ids like openai/ft:...) still resolve directly.

Providers resolved directly by prefix:

Prefix	Key	Endpoint
anthropic/*	ANTHROPIC_API_KEY	Anthropic (OpenAI-compat)
openai/*	OPENAI_API_KEY	OpenAI
google/*	GEMINI_API_KEY	Google Gemini (OpenAI-compat)
x-ai/* or xai/*	XAI_API_KEY	xAI
groq/*	GROQ_API_KEY	Groq (rest of the id passed through, slashes and all)
mistralai/*	MISTRAL_API_KEY	Mistral
deepseek/*	DEEPSEEK_API_KEY	DeepSeek

The wire format defaults to Chat Completions (-P wire=chat) for broad OpenAI-compatible endpoint support. Switch to -P wire=responses when a direct OpenAI or compatible endpoint requires the Responses API.

How it works

Motion tool calls state where to go, not how long to move. For absolute modes, the move tool (move_joints for joint spaces, move_to for Cartesian pose modes) interpolates named partial targets from the observed state at a fixed safe speed. The default max_speed_frac=0.1 allows a tenth of each dimension's range per second, subject to a 5%-of-range per-step ceiling that matches the core's default delta backstop. At that default a near-full-range move exceeds the 10 s per-call playout cap, so the agent receives a split-the-move error and issues it as two smaller motions; raise the fraction (up to 0.5 before the ceiling binds at 10 Hz) for faster arms. The tool result reports the computed step count and, when the embodiment declares control_hz, the corresponding playout time. duration_s is not part of either motion tool.

Every move tool call also requires a note with one or two plain sentences describing the current observation and why the agent chose that motion. The user reads these notes live and in the saved transcript to follow what the agent sees and decides.

For displacement modes, move_by splits the requested total so every action fits the box side in that direction. The action box is the embodiment author's per-step speed statement, so max_speed_frac does not apply to displacement modes. done and give_up end the trial through the core's policy-stop channel.

When control_hz is None, the plugin uses a 10 Hz fallback to compute step counts and the per-call playout cap, but leaves the emitted chunk rate unset. The embodiment then plays the chunk at its native rate. In this case the speed and playout caps are step-count constructs, not wall-clock guarantees, and the tool result does not report seconds.

When the embodiment publishes operating notes via EmbodimentInfo.docs (joint layout, sign conventions, gripper polarity), the policy appends them to the system prompt as an Embodiment notes: section. The per-step observation also labels the proprioceptive state vector with the action dimension names (left_j0=0.01 ...) whenever the mapping is unambiguous.

Every action still passes the CLI's default safety approvers (bounds clamp plus per-step delta limit); the plugin contains no safety-critical code path of its own. An explicit --max-action-delta tighter than 5% of range can truncate absolute interpolants. In displacement modes, a value tighter than the action box can truncate each move_by step. Either setting can make the executed motion fall short of the tool's requested total.

Warning: Guardrails are on by default at the CLI. Never pass --disable-guardrails on real hardware unless you fully trust the policy and the rig.

Configuration knobs (all -P key=value): model, base_url, api_key_env, wire, max_llm_calls (default 100), temperature, effort, max_speed_frac, transcript_echo. Set -P transcript_echo=true to print live [agent] conversation lines to stderr, including goals, observation summaries, assistant output, tool calls, and tool results. Move notes appear inside the echoed tool-call arguments. The speed fraction defaults to 0.1 and applies only to absolute modes.

LLMAgentPolicy.transcript() returns the current conversation as a deep copy with streamed camera frames replaced by omission markers, ready for core eval-log persistence. Camera labels such as camera 'top_cam' (step 480): provide the join key from a transcript observation to its stored frame. Live Rerun transcript streaming happens automatically when a Rerun sink is attached.

Reasoning effort defaults to low: robot control is latency-sensitive (the arm stands still while the model thinks), safety guardrails sit below the model either way, and frontier models at low effort remain strong at this task shape. Raise it for hard manipulation problems (-P effort=high) or pass -P effort=none to omit the parameter for endpoints that reject it (the CLI reads a bare none as null). To send the literal wire value none and disable reasoning, quote it: -P effort="'none'". GPT-5.x on chat completions requires the literal none when function tools are in play (any other value, or omitting the field, is a 400). In Python, effort=None omits the field and effort="none" sends the wire value.

Reasoning effort on OpenAI models

Recent OpenAI reasoning models can reject function tools on the Chat Completions wire with an error like this:

Function tools with reasoning_effort are not supported. To use function
tools, use /v1/responses or set reasoning_effort to 'none'.

This is a Chat Completions API restriction, not an inspect-robots bug. Use a direct OpenAI endpoint and select the Responses wire to keep reasoning enabled:

inspect-robots "pick up the cube" --policy agent \
    -P model=openai/gpt-5.6-sol -P wire=responses -P effort=medium \
    --embodiment cubepick