cua-speedrun

Compare computer-use agents by performance, time, and cost on real desktop tasks.

Quickstart

Install and configure your credentials:

pip install cua-speedrun
cua-speedrun setup
cua-speedrun benchmark

Choose an agent template and benchmark, review or edit the configuration, then start. The terminal shows preparation progress and opens the evaluation dashboard. setup lets you add Modal credentials, API keys, and other variables.

Prebuilt desktop images are imported from Docker Hub and cached in Modal. Models and application setup are cached on first use. Data lives in ~/.local/share/cua-speedrun; set CUA_SPEEDRUN_HOME to use another location. Use cua-speedrun doctor to inspect your installation.

Run an evaluation

Evaluations run on Modal by default, using credentials from setup, your Modal CLI profile, or environment variables. For an API agent:

export ANTHROPIC_API_KEY="your-api-key"
cua-speedrun benchmark --dataset osworld-50 --agent claude

For Claude Code, use --agent claude_code with ANTHROPIC_API_KEY or CLAUDE_CODE_OAUTH_TOKEN from claude setup-token. If both are set, the OAuth token is used. Set CLAUDE_CODE_MODEL and CLAUDE_CODE_EFFORT, and pass them with --env CLAUDE_CODE_MODEL --env CLAUDE_CODE_EFFORT to select the model and effort.

Arguments prefill the wizard. Add --no-input to run them directly. For scripts and AI agents, --json skips the wizard and writes JSON records to stdout; preparation logs go to stderr:

cua-speedrun --json benchmark --dataset osworld-50 --agent qwen3vl

The command follows progress until the evaluation finishes; add --background to return a JSON evaluation ID after preparation and queuing. --json also works with setup, catalog, status, and evaluations. To launch and inspect evaluations in a browser, run cua-speedrun dashboard.

• Add --parallel-evaluations 4 to run four agent replicas in parallel.
• Bundled GPU agents select their GPU automatically; override it with --gpu L40S.
• Add --no-preload to disable environment preloading.
• To use local Linux hardware, add --compute local --environment local. Local desktops require KVM/QEMU.

Inspect results or download trajectories:

cua-speedrun evaluations
cua-speedrun status RUN_ID
cua-speedrun export RUN_ID

Use cua-speedrun catalog to list available agents and benchmarks, or cua-speedrun help benchmark for more options.

Benchmarks

The offline variants contain the full offline task sets; the smaller variants are representative subsets. MyPCBench also requires a MYPCBENCH_JUDGE_API_KEY for its evaluator; see its setup instructions.

Bring your own agent

Start from an implementation in agents/. Each agent has two files:

• init.py prepares dependencies or starts a model server before task timing begins.
• agent.py receives the environment URL and task description, then interacts through Computer.

Submit the folder directly:

cua-speedrun validate --agent ./my-agent
cua-speedrun benchmark --dataset osworld-50 --agent ./my-agent

Additional packages can be installed by init.py; the submission uploads init.py and agent.py. An optional agent.json declares gpu and required_environment_variables. Use optional_environment_variables for variables forwarded when set. To contribute an agent, add its folder to agents/; it is discovered automatically.

Bring your own benchmark

A benchmark is a folder with a manifest.yaml, task folders containing task.yaml, and its environment setup and verifier. The manifest lists tasks:

name: my-benchmark
version: "1"
tasks: [tasks/my-task]

Each task.yaml specifies task_id, description, and an env mapping:

task_id: my-task
description: The task for the agent to complete.
env:
  kind: gym-anything
  env_dir: ${BENCHMARK_DIR}/environment
  task_id: my-task

Keep the Gym-Anything environment and its task setup/verifier inside the benchmark folder. Then:

cua-speedrun validate --dataset ./my-benchmark
cua-speedrun benchmark --dataset ./my-benchmark --agent ./my-agent

The benchmark is copied into the installation. Increase its version when changing a registered task set. To contribute it, add the folder under benchmarks/ and its name to catalog/benchmarks.yaml; packaging is automatic.

Repository structure

• agents/ — agent implementations.
• benchmarks/ — task sets and benchmark definitions.
• src/cua_speedrun/ — execution, timing, scoring, and the dashboard.
• src/cua_speedrun/compute_runners/ — local and Slurm runners.