cua-speedrun

Compare computer-use agents by performance, time, and cost on real desktop tasks.

Quickstart

Install and configure your credentials:

pip install cua-speedrun
cua-speedrun setup
cua-speedrun benchmark --dataset osworld-50 --agent qwen3vl

Prebuilt desktop images are imported from Docker Hub and cached in Modal. Models and application setup are cached on first use. Data lives in ~/.local/share/cua-speedrun; set CUA_SPEEDRUN_HOME to use another location. Use cua-speedrun doctor to inspect your installation.

Run an evaluation

Evaluations run on Modal by default, using credentials from setup, your Modal CLI profile, or environment variables. For an API agent:

export ANTHROPIC_API_KEY="your-api-key"
cua-speedrun benchmark --dataset osworld-50 --agent claude

The command follows progress until the evaluation finishes. To launch and inspect evaluations in a browser, run cua-speedrun dashboard.

• Add --parallel-evaluations 4 to run four agent replicas in parallel.
• Bundled GPU agents select their GPU automatically; override it with --gpu L40S.
• Add --no-preload to disable environment preloading.
• To use local Linux hardware, add --compute local --environment local. Local desktops require KVM/QEMU.

Inspect results or download trajectories:

cua-speedrun evaluations
cua-speedrun status RUN_ID
cua-speedrun export RUN_ID

Use cua-speedrun catalog to list available agents and benchmarks, or cua-speedrun help benchmark for more options.

Benchmarks

The offline variants contain the full offline task sets; the smaller variants are representative subsets. MyPCBench also requires a MYPCBENCH_JUDGE_API_KEY for its evaluator; see its setup instructions.

Bring your own agent

Start from an implementation in agents/. Each agent has two files:

• init.py prepares dependencies or starts a model server before task timing begins.
• agent.py receives the environment URL and task description, then interacts through Computer.

Submit the folder directly:

cua-speedrun validate --agent ./my-agent
cua-speedrun benchmark --dataset osworld-50 --agent ./my-agent

Additional packages can be installed by init.py; the submission uploads init.py and agent.py. An optional agent.json declares gpu and required_environment_variables. To contribute an agent, add its folder to agents/; it is discovered automatically.

Bring your own benchmark

A benchmark is a folder with a manifest.yaml, task folders containing task.yaml, and its environment setup and verifier. The manifest lists tasks:

name: my-benchmark
version: "1"
tasks: [tasks/my-task]

Each task.yaml specifies task_id, description, and an env mapping:

task_id: my-task
description: The task for the agent to complete.
env:
  kind: gym-anything
  env_dir: ${BENCHMARK_DIR}/environment
  task_id: my-task

Keep the Gym-Anything environment and its task setup/verifier inside the benchmark folder. Then:

cua-speedrun validate --dataset ./my-benchmark
cua-speedrun benchmark --dataset ./my-benchmark --agent ./my-agent

The benchmark is copied into the installation. Increase its version when changing a registered task set. To contribute it, add the folder under benchmarks/ and its name to catalog/benchmarks.yaml; packaging is automatic.

Repository structure

• agents/ — agent implementations.
• benchmarks/ — task sets and benchmark definitions.
• src/cua_speedrun/ — execution, timing, scoring, and the dashboard.
• src/cua_speedrun/compute_runners/ — local and Slurm runners.