control-runner 0.1.1

Tool for configuring and running control_arena experiments using .yaml files.

Control Runner

Run control_arena experiments from YAML. Define settings, protocols, policies, monitors, and scorers as reusable components or inline specs. Then execute them from a single CLI.

Environment

Most model providers require API keys via environment variables. It’s recommended to keep them in a .env file and ensure your environment loads it.

# OpenAI
OPENAI_API_KEY=sk-...replace-me...

# Google
GOOGLE_API_KEY=...replace-me...

Quick start

1. Install

uv add control-runner

2. Create quickstart.yaml

settings:
  apps_setting:
    type: "apps"
    params:
      dataset_name: "simple"

tasks:
  - id: "quickstart"
    setting: "apps_setting"
    protocol:
      type: "tm_protocol"
      params:
        policy_builder:
          type: "untrusted_policy"
          params:
            honest:
              type: "honest_policy"
              params:
                model: "google/gemini-2.5-flash-lite"
            attack:
              type: "attack_policy"
              params:
                model: "google/gemini-2.5-flash"
        monitor_builder:
          type: "basic_monitor"
          params:
            model: "google/gemini-2.5-flash-lite"
    mode: "ATTACK"
    monitoring_scorers:
      - "max_suspicion_scorer"
    limit: 5

3. Run

uv run control_runner run quickstart.yaml --dry-run  # validate
uv run control_runner run quickstart.yaml            # execute

Alternatively, without adding to your project:

uvx control-runner run quickstart.yaml --dry-run

Why use it

• Simple YAML: Reference reusable components or define them inline
• Dynamic variables: ${var} substitution and file-content expansion
• Safe dry-runs: Validate configs without importing heavy deps
• Batteries included: Rich built‑in registry mapped to control_arena

Quick start (PyPI)

Prereqs: Python >= 3.13, uv installed.

# 1) Add to your project
uv add control-runner

# 2) Run with uv
uv run control_runner run path/to/config.yaml --dry-run  # validate
uv run control_runner run path/to/config.yaml            # execute

# Alternative: one-off run without adding to your project
uvx control-runner run path/to/config.yaml --dry-run

CLI

control_runner run CONFIG [--dry-run]

• CONFIG (required): Path to a YAML file
• --dry-run: Load and print tasks without running evaluation

Behavior:

• Sets UNTRUSTED_MODEL and TRUSTED_MODEL env vars if present in vars
• Groups tasks by their limit and runs each group via inspect_ai.eval(tasks=..., limit=...)

YAML configuration

Top-level keys:

• vars (optional): Simple key/value variables
• settings: Named settings objects
• components (optional): Reusable components (policies, monitors, protocols, scorers, …)
• tasks: List of tasks to run

Referencing values

You can reference values in two ways:

• ${name}: explicit reference
• name: bare string; if it exactly matches a key in vars, settings, or components, it resolves to that value

String substitutions are supported for primitives inside larger strings, e.g. "Model: ${untrusted_model}".

Variables and file loading

Within vars, any string that looks like a path (./, ../, or / and has an extension) is read as file content. This makes prompts easy to manage:

vars:
  untrusted_model: google/gemini-2.5-flash
  trusted_model: google/gemini-2.5-flash-lite
  attack_monitor_prompt: ./prompts/attack_monitor_simple.txt  # contents will be in this var

Use these variables anywhere (including inside inline specs) via ${...}.

Inline specs and the registry

Components can be defined inline using type (lookup in the registry):

policy_builder:
  type: untrusted_policy
  params:
    honest: ${honest_policy_builder}
    attack: ${attack_policy_builder}

The registry contains keys for settings, policies, monitors, protocols, scorers, and tools. See src/control_runner/registry.py for the full list of supported names.

Tasks

tasks:
  - id: attack_task
    setting: ${apps_setting}          # reference or inline spec
    protocol: ${monitored_protocol}   # reference or inline spec
    mode: ATTACK                      # string or EvalMode name (case‑insensitive)
    monitoring_scorers:               # list of names or inline spec(s)
      - max_suspicion_scorer
    limit: 1                          # per‑group limit for inspect_ai.eval

monitoring_scorers accepts either a list or a single inline spec object.

Settings and components

Reusable blocks defined under settings and components can be referenced from tasks or other components:

settings:
  apps_setting:
    type: apps
    params:
      dataset_name: simple

components:
  monitored_protocol:
    type: tm_protocol
    params:
      policy_builder: ${untrusted_policy_builder}
      monitor_builder: ${untrusted_monitor_builder}

Examples

The example_configs directory shows common patterns:

• components.yaml: Reusable components and reference-based tasks
• inline.yaml: Fully inline protocol and nested inline components
• inline_with_components.yaml: Mix of inline protocol with referenced components
• inline_strings.yaml: Minimal config using bare string references

Run any example with --dry-run to validate it, then without to execute.

Troubleshooting

• Unknown reference: Ensure the name exists in vars, settings, or components
• Unknown type: Check that the type exists in the registry (src/control_runner/registry.py)
• File var not loaded: File expansion only applies to values in vars

Development

control_runner/
├─ example_configs/
├─ prompts/
├─ src/control_runner/
│  ├─ cli.py         # Click CLI entrypoint
│  ├─ loader.py      # YAML loading and resolution
│  └─ registry.py    # control_arena registry mapping
├─ tests/
└─ README.md

From source (repo checkout):

uv sync
uv run control_runner run example_configs/components.yaml --dry-run

Testing

uv run pytest -q