Skip to main content

cua-bench

Run computer-use benchmark datasets on sandboxes.

Cua Cloud has ended; --on cloud now fails with a pointer to the notice.

Documentation - Installation, guides, and API reference.

Registry tasksets

Browse the tasksets at cua.ai/cuabench/registry and run one by name on a desktop sandbox:

cb dataset list                                       # tasksets and versions
cb run cua-bench-basic --task-filter click-button     # oracle check, local desktop
cb run cua-bench-basic --agent cua-agent --model anthropic/claude-sonnet-4-20250514 -j 4

A name (or name@version) resolves through the bundled index to a pinned commit, fetched once into ~/.cua/cbregistry. CUA_BENCH_REGISTRY points at another index.

Running tasks

One command runs a task or a dataset on this machine. There is no pool or provider to set up.

cb run example_tasks/hello_file_env                   # local gVisor container (default)
cb run datasets/cua-bench-basic -j 8                  # 8 at a time, locally
cb run my_task --agent cua-agent --model anthropic/claude-sonnet-4-20250514
cb run my_task --dry-run                              # what would run, nothing starts

cb run <path> runs one task (a directory with main.py, variant --variant-id) or a dataset (every task and variant). It runs in the foreground with per-variant status; add --detach to return at once and follow with cb run watch <id>.

Flag Meaning
--on local Where. local: sandboxes on this machine through the cua SDK. cloud is removed (Cua Cloud has ended). Default: CUA_DEFAULT_ON, then cua config set default.on, then local.
--kind auto|container|vm What kind. auto (default): Linux runs as a container unless the task or the image's variant index says VM; Windows, macOS and Android are VM-only. Default: CUA_DEFAULT_KIND, then default.kind.
--runtime auto|gvisor|runc|qemu|lume Which engine, and it implies the kind. Local containers: gvisor, runc; local VMs: qemu, lume. auto (default) lets the SDK pick. A combination that does not exist is an error listing the valid values. Default: CUA_DEFAULT_RUNTIME, then default.runtime (skipped for a task it does not fit).
--image REF Image for every task: an OS alias (linux, windows, macos:tahoe), ghcr.io/trycua/<os>, any registry ref or a digest. Otherwise the task's setup_config.image, then CUA_BENCH_IMAGE, then the canonical image for the OS.
--cpu, --memory Resources per sandbox (--memory 4G).
-j, --max-parallel Variants at once.
--attempts N Run every variant N times (fresh sandbox each); summary.json reports pass@k.
--retries N Retry a variant that failed before evaluation (sandbox start, connection), with backoff.
--dry-run Print each variant's image, kind, variant and backend, then exit.

Deprecated 0.2.11 flags still parse, with a notice naming the new ones: --platform linux-docker (--kind container), linux-qemu (--kind vm --runtime qemu --image linux), windows-qemu (--kind vm --runtime qemu --image windows), android-qemu (--kind vm --image android), all local; --provider-type native has no effect.

Results land in ~/.local/share/cua-bench/runs/<run id>/<task>_v<n>/ (run.log, result.json, task_<n>_trace/, trajectory.json) with a summary.json per run. cb run list | info | watch | logs | stop inspect runs; cb dataset build <run dir> exports traces (aguvis-stage-1, gui-r1).

  • result.json records what ran (on, backend such as local-gvisor, the sandbox ref such as local:<name>, kind, runtime (the engine, when one was chosen), image_ref, image_variant, image_digest as the pinned repo@sha256, arch) and carries Harbor's trial fields (task_name, trial_name, verifier_result.rewards, exception_info, agent_info, per-phase timing).
  • trajectory.json is an ATIF-v1.8 trajectory (Harbor's format) with screenshots under imgs/.
  • summary.json adds pass_at_k (with --attempts), stats and a targets breakdown. New keys are additive; schema_version is 2.

Datasets

cb run <name>[@version] runs a dataset from the registry. Each version is pinned to a git commit and fetched once (shallow and sparse) into ~/.cua/cbregistry; a plain name means the latest version. cb dataset list shows them. CUA_BENCH_REGISTRY points at another index (a path or URL), including a Harbor registry.json.

Images

Tasks run on the canonical ghcr.io/trycua/{linux,windows,macos} images unless they name another registry image. Each canonical image is one index with a rootfs (containers) and a -disk containerDisk (VMs); cb image list shows them.

Tasks

A task declares its environment in computer:

cb.Task(
    description="...",
    computer={
        "provider": "native",  # a real sandbox (the only provider since 0.3)
        "setup_config": {
            "os_type": "linux",  # linux | windows | macos (local only) | android (local only)
            "image": "ghcr.io/me/my-desktop:docker-1",  # optional
            "kind": "container",  # optional preference: container | vm
            "kinds": ["vm"],  # optional requirement (e.g. the task needs a kernel)
            "server_port": 8000,  # optional: the image's own daemon, used for readiness
        },
    },
)

Setup, the oracle or agent, and evaluate run in the cb process and drive the sandbox through cua-spacesd (shell, files, screen, input), so tasks that use them need an image with cua-spacesd. Agents are loaded in-process (--agent or --agent-import-path). cb task create <dir> scaffolds a Linux container task.

A coding-agent harness as the agent

--agent harness runs a coding-agent harness (Claude Code, Codex, Gemini CLI, OpenCode, Goose, ...) inside the task's own sandbox through the cua SDK's agents API: "harness X in image Y", with the image chosen as above. The harness gets the task description as its prompt and the sandbox's own MCP tools (screen, input, windows); the task's evaluate scores the sandbox afterwards.

CUA_BENCH_HARNESS=claude-code CUA_BENCH_HARNESS_KEYS=ANTHROPIC_API_KEY \
  cb run example_tasks/hello_file_env --agent harness

Settings are environment variables (or constructor kwargs): CUA_BENCH_HARNESS (cua agent harnesses lists them), CUA_BENCH_HARNESS_KEYS (provider key variables forwarded from this process), CUA_BENCH_HARNESS_MODEL (or --model), CUA_BENCH_HARNESS_BASE_URL (a proxy or compatible endpoint), CUA_BENCH_HARNESS_MCP (NAME=URL,...), CUA_BENCH_HARNESS_TIMEOUT (seconds, default 1800). The session must be a cua-spacesd sandbox; for a legacy computer-server session set CUA_BENCH_SPACESD_URL and CUA_BENCH_SPACESD_TOKEN. Events and the result land in the task's agent log directory (harness-events.jsonl, harness-result.json).

Upgrading from 0.2

  • The simulated (Playwright) provider is gone. A task that still declares provider: "simulated" runs on the Linux container with a deprecation warning (CUA_BENCH_STRICT=1 makes it an error). HTML pages opened with session.launch_window(html=...) need bench-ui in the image.
  • cb image create|shell|clone|delete, cb platform, cb prune --docker and cb agent build|push are deprecated no-ops; use --image <registry ref>.
  • RL workers, the dataloader and trainers moved to cua-bench-rl (cua_bench.workers still imports, with a warning).
  • RemoteDesktopSession keeps 0.2.7 semantics for code written against it (for example Agents' Last Exam): run_command reports return_code 0 unless strict_exit_codes=True, there is no shell timeout unless timeout= is set, api_url sessions speak to computer-server (or cua-spacesd: port 3211 or a sandbox ref such as local:<name>, direct:<host:port>), and session.interface keeps the cua-computer method set. scripts/ale_compat/check_ale.py checks ALE against a checkout.

Running Tests

uv sync --extra dev
uv run pytest -q cua_bench/tests --ignore=cua_bench/tests/live

The suite is hermetic: cua-sandbox is replaced by an in-memory sandbox (cua_bench/tests/fakes.py) and a loopback computer-server stand-in, so nothing starts a container or VM. It includes the CLI parser golden (tests/golden/cli_tree.json), export schema snapshots and the ALE compatibility tests. After a deliberate, additive CLI or schema change, regenerate the goldens with CUA_BENCH_UPDATE_GOLDENS=1 and review the diff.

Opt-in live tests (they start real sandboxes):

CUA_BENCH_E2E_LOCAL=1 uv run pytest -s cua_bench/tests/live/test_local_live.py

The ALE check runs agents-last-exam's own loader and harness against this cua-bench:

python scripts/ale_compat/check_ale.py --ale ../agents-last-exam --level 3

Metadata

Release files for cua-bench 0.4.9

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for cua-bench 0.4.9
File Size Uploaded Provenance Actions
cua_bench-0.4.9.tar.gz 245.5 kB Without provenance Details for cua_bench-0.4.9.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for cua-bench 0.4.9
File Interpreter ABI Platform Actions
cua_bench-0.4.9-py3-none-any.whl Python 3 none any Details for cua_bench-0.4.9-py3-none-any.whl

Total release size: 526.7 kB

Release files / cua_bench-0.4.9.tar.gz

Download URL cua_bench-0.4.9.tar.gz
Size 245.5 kB
Tags Source
SHA-256 checksum
How to use checksums
4ba35b48d6ca55b60743259b924ec5ba3660d63c40697775b053b2d75f5fe5ec
BLAKE2b-256 checksum
How to use checksums
bcadf15f8c9b19f9aa1b7608382e6f5904e57b898ea0ee0d8f6fa56543d1f66b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.9

Release files / cua_bench-0.4.9-py3-none-any.whl

Download URL cua_bench-0.4.9-py3-none-any.whl
Size 281.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b590f236898054c04c9b95f808da817235bb0e63a535520471a1b7cf67b76c4f
BLAKE2b-256 checksum
How to use checksums
92b48ff08fa30f5ad953d448f354582ab5b8fc0aa88b6c765b5d1fd000c5767d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.9

Security for cua-bench 0.4.9

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page