rolloutdev 0.2.0

Python SDK + CLI for the rollout agent-environment platform.

rollout — Python SDK + CLI

Author agent environments in Python, push them to the rollout platform, and run the closed loop (rollouts → verifier → fine-tune → compare) from your terminal.

Install

pip install rolloutdev

Note: the package is published as rolloutdev on PyPI because rollout was taken in 2011 by an unrelated, unmaintained library. The Python module name is still rollout, so from rollout import … works as expected. We've filed a PEP 541 claim to recover the name.

Authenticate

1. Mint an API key at https://rollout.mv37.org/settings/api-keys.
2. rollout login and paste it.

$ rollout login
Get an API key at https://rollout.mv37.org/settings/api-keys
Paste API key (rk_…): ********
✓ saved credentials for https://rollout.mv37.org

The key lives in ~/.config/rollout/credentials (mode 0600). Override per-call with ROLLOUT_API_KEY / ROLLOUT_ENDPOINT env vars or pass them to Client().

Define an environment

# refund_agent.py
from rollout import Environment, Task

env = Environment(
    slug="refund-agent",
    name="Refund Agent",
    template="workflow",
    description="Customer-support refund workflow",
    config={
        "tools": [
            {
                "name": "lookup_order",
                "kind": "lookup",
                "fixturePath": "orders.${args.order_id}",
            },
            {
                "name": "refund",
                "kind": "action",
                "appendTo": "refunds",
                "value": {
                    "order_id": "${args.order_id}",
                    "amount": "${args.amount}",
                },
            },
        ],
        "fixtures": {
            "orders": {
                "O-1": {"customer": "Alice", "amount": 100, "status": "shipped"},
            },
        },
        "initialMutableState": {"refunds": []},
    },
    tasks=[
        Task(
            instruction="Refund order O-1 for $100 for Alice",
            hidden_ground_truth={
                "refunds": [{"order_id": "O-1", "amount": 100.0}],
            },
        ),
    ],
)

For larger task sets, load from JSONL:

env.tasks = Task.from_jsonl("tasks.jsonl")

Each line: {"instruction": "...", "hidden_ground_truth": {...}, "split": "dev"}.

Push, run, improve

# Upload the env
rollout push refund_agent.py
# → https://rollout.mv37.org/environments/refund-agent

# Run a baseline
rollout run refund-agent --model openai/gpt-4o-mini --n 50

# List recent runs
rollout runs

# Fine-tune on the passing rollouts
rollout improve --run <run-id> --model openai/gpt-4o-mini-2024-07-18 \
                --method sft --suffix v1

# Track training
rollout jobs
rollout job <job-id>

DPO instead of SFT (Together AI only):

rollout improve --run <run-id> --model together/deepseek-ai/DeepSeek-V3.1 \
                --method dpo

Programmatic use

The CLI is a thin wrapper over the Client:

from rollout import Client

c = Client()
run = c.create_run(env_slug="refund-agent", model="openai/gpt-4o-mini", n=50)
print(run["runId"])

# When the run completes, kick off SFT on its passing rollouts
ids = c.passing_rollout_ids(run["runId"])
job = c.create_training_job(
    source_run_id=run["runId"],
    rollout_ids=ids,
    base_model="openai/gpt-4o-mini-2024-07-18",
    method="sft",
    suffix="v1",
)
print(job["id"])

Templates

Template	What the config holds
sql	schema (DDL string) + optional seed (INSERT SQL). Tools are fixed: describe_schema, run_sql, submit_answer.
workflow	tools[] (lookup / action / mock kinds), fixtures{} (read-only), initialMutableState{} (state arrays the agent can append to).
code-editing	files{} (path → content). Verifier checks final file state against task ground truth.
custom	tools[] with paramsSchema + templated response + optional state mutates. The escape hatch.
browser	pages[] with elements + initialUrl. Mock DOM for navigation/form tasks.

The exact JSON shapes match what the dashboard's "New environment" form expects — those defaults are the easiest reference.

Local rollouts (rollout test)

Run a single rollout locally, no platform round-trip — fastest way to iterate on an environment definition.

rollout test my_env.py --model openai/gpt-4o-mini
rollout test my_env.py --model openai/gpt-4o --task 0 --temperature 0.0

The local engine reads your Environment, calls the model directly (through the AI Gateway, or direct OpenAI for openai/* models), executes the env's declarative tools in-process, and runs a subset of verifier primitives:

• workflow / custom / code-editing templates: ✅ supported
• sql / browser templates: not yet (need sqlite + DOM runtimes)
• verifier primitives: state_match, contains_terms, tool_was_called, no_tool_errors

Output is a per-task pass/fail with the trace of tool calls. Push to the platform once you're happy with the env shape.

Template helpers

The SDK ships builders so you don't hand-roll the platform's JSON shapes:

from rollout import (
    Environment, Task,
    lookup_tool, action_tool, mock_tool,
    custom_tool, custom_config,
    workflow_config, code_editing_config,
    browser_page, browser_config,
    primitive, verifier_config,
)

env = Environment(
    slug="refund-agent",
    name="Refund Agent",
    template="workflow",
    config=workflow_config(
        tools=[
            lookup_tool("get_order", from_array="orders", match_field="id",
                        description="Read an order.", params_schema={"id": "string"}),
            action_tool("refund", state_array="refunds",
                        description="Issue a refund.",
                        params_schema={"order_id": "string", "amount": "number"}),
        ],
        fixtures={"orders": [{"id": "O-1", "customer": "Alice", "amount": 100}]},
        initial_mutable_state={"refunds": []},
    ),
    tasks=[
        Task(instruction="Refund order O-1 for $100",
             hidden_ground_truth={"refunds": [{"order_id": "O-1", "amount": 100}]}),
    ],
)

env.verifier = verifier_config(
    primitive("called_lookup", "tool_was_called", tool="get_order", weight=1),
    primitive("refunds_count", "state_match",
              expected={"refunds.length": 1}, weight=2),
    primitive("no_errors", "no_tool_errors", weight=1),
)

Status

v0.2 — adds template helpers, rollout test local rollouts, and the compare CLI command. Real test execution for the code-editing template (running npm test in a sandbox) is on the v0.3 roadmap; needs sandbox infrastructure (Vercel Sandbox or Modal).