cua-bench
Run computer-use benchmark datasets on sandboxes.
Cua Cloud has ended; --on cloud now fails with a pointer to the notice.
Documentation - Installation, guides, and API reference.
Registry tasksets
Browse the tasksets at cua.ai/cuabench/registry and run one by name on a desktop sandbox:
cb dataset list # tasksets and versions
cb run cua-bench-basic --task-filter click-button # oracle check, local desktop
cb run cua-bench-basic --agent cua-agent --model anthropic/claude-sonnet-4-20250514 -j 4
A name (or name@version) resolves through the bundled index to a pinned commit, fetched
once into ~/.cua/cbregistry. CUA_BENCH_REGISTRY points at another index.
Running tasks
One command runs a task or a dataset on this machine. There is no pool or provider to set up.
cb run example_tasks/hello_file_env # local gVisor container (default)
cb run datasets/cua-bench-basic -j 8 # 8 at a time, locally
cb run my_task --agent cua-agent --model anthropic/claude-sonnet-4-20250514
cb run my_task --dry-run # what would run, nothing starts
cb run <path> runs one task (a directory with main.py, variant --variant-id) or a
dataset (every task and variant). It runs in the foreground with per-variant status; add
--detach to return at once and follow with cb run watch <id>.
| Flag | Meaning |
|---|---|
--on local |
Where. local: sandboxes on this machine through the cua SDK. cloud is removed (Cua Cloud has ended). Default: CUA_DEFAULT_ON, then cua config set default.on, then local. |
--kind auto|container|vm |
What kind. auto (default): Linux runs as a container unless the task or the image's variant index says VM; Windows, macOS and Android are VM-only. Default: CUA_DEFAULT_KIND, then default.kind. |
--runtime auto|gvisor|runc|qemu|lume |
Which engine, and it implies the kind. Local containers: gvisor, runc; local VMs: qemu, lume. auto (default) lets the SDK pick. A combination that does not exist is an error listing the valid values. Default: CUA_DEFAULT_RUNTIME, then default.runtime (skipped for a task it does not fit). |
--image REF |
Image for every task: an OS alias (linux, windows, macos:tahoe), ghcr.io/trycua/<os>, any registry ref or a digest. Otherwise the task's setup_config.image, then CUA_BENCH_IMAGE, then the canonical image for the OS. |
--cpu, --memory |
Resources per sandbox (--memory 4G). |
-j, --max-parallel |
Variants at once. |
--attempts N |
Run every variant N times (fresh sandbox each); summary.json reports pass@k. |
--retries N |
Retry a variant that failed before evaluation (sandbox start, connection), with backoff. |
--dry-run |
Print each variant's image, kind, variant and backend, then exit. |
Deprecated 0.2.11 flags still parse, with a notice naming the new ones: --platform linux-docker (--kind container), linux-qemu (--kind vm --runtime qemu --image linux), windows-qemu (--kind vm --runtime qemu --image windows), android-qemu
(--kind vm --image android), all local; --provider-type native has no effect.
Results land in ~/.local/share/cua-bench/runs/<run id>/<task>_v<n>/ (run.log,
result.json, task_<n>_trace/, trajectory.json) with a summary.json per run.
cb run list | info | watch | logs | stop inspect runs; cb dataset build <run dir>
exports traces (aguvis-stage-1, gui-r1).
result.jsonrecords what ran (on,backendsuch aslocal-gvisor, thesandboxref such aslocal:<name>,kind,runtime(the engine, when one was chosen),image_ref,image_variant,image_digestas the pinnedrepo@sha256,arch) and carries Harbor's trial fields (task_name,trial_name,verifier_result.rewards,exception_info,agent_info, per-phase timing).trajectory.jsonis an ATIF-v1.8 trajectory (Harbor's format) with screenshots underimgs/.summary.jsonaddspass_at_k(with--attempts),statsand atargetsbreakdown. New keys are additive;schema_versionis 2.
Datasets
cb run <name>[@version] runs a dataset from the registry. Each version is pinned to a
git commit and fetched once (shallow and sparse) into ~/.cua/cbregistry; a plain name
means the latest version. cb dataset list shows them. CUA_BENCH_REGISTRY points at
another index (a path or URL), including a Harbor registry.json.
Images
Tasks run on the canonical ghcr.io/trycua/{linux,windows,macos} images unless they name
another registry image. Each canonical image is one index with a rootfs (containers) and
a -disk containerDisk (VMs); cb image list shows them.
Tasks
A task declares its environment in computer:
cb.Task(
description="...",
computer={
"provider": "native", # a real sandbox (the only provider since 0.3)
"setup_config": {
"os_type": "linux", # linux | windows | macos (local only) | android (local only)
"image": "ghcr.io/me/my-desktop:docker-1", # optional
"kind": "container", # optional preference: container | vm
"kinds": ["vm"], # optional requirement (e.g. the task needs a kernel)
"server_port": 8000, # optional: the image's own daemon, used for readiness
},
},
)
Setup, the oracle or agent, and evaluate run in the cb process and drive the sandbox
through cua-spacesd (shell, files, screen, input), so tasks that use them need an image
with cua-spacesd. Agents are loaded in-process (--agent or --agent-import-path).
cb task create <dir> scaffolds a Linux container task.
A coding-agent harness as the agent
--agent harness runs a coding-agent harness (Claude Code, Codex, Gemini CLI, OpenCode,
Goose, ...) inside the task's own sandbox through the cua SDK's agents API: "harness X in
image Y", with the image chosen as above. The harness gets the task description as its
prompt and the sandbox's own MCP tools (screen, input, windows); the task's evaluate
scores the sandbox afterwards.
CUA_BENCH_HARNESS=claude-code CUA_BENCH_HARNESS_KEYS=ANTHROPIC_API_KEY \
cb run example_tasks/hello_file_env --agent harness
Settings are environment variables (or constructor kwargs): CUA_BENCH_HARNESS
(cua agent harnesses lists them), CUA_BENCH_HARNESS_KEYS (provider key variables
forwarded from this process), CUA_BENCH_HARNESS_MODEL (or --model),
CUA_BENCH_HARNESS_BASE_URL (a proxy or compatible endpoint),
CUA_BENCH_HARNESS_MCP (NAME=URL,...), CUA_BENCH_HARNESS_TIMEOUT (seconds, default
1800). The session must be a cua-spacesd sandbox; for a legacy computer-server session set
CUA_BENCH_SPACESD_URL and CUA_BENCH_SPACESD_TOKEN. Events and the result land in the
task's agent log directory (harness-events.jsonl, harness-result.json).
Upgrading from 0.2
- The simulated (Playwright) provider is gone. A task that still declares
provider: "simulated"runs on the Linux container with a deprecation warning (CUA_BENCH_STRICT=1makes it an error). HTML pages opened withsession.launch_window(html=...)need bench-ui in the image. cb image create|shell|clone|delete,cb platform,cb prune --dockerandcb agent build|pushare deprecated no-ops; use--image <registry ref>.- RL workers, the dataloader and trainers moved to
cua-bench-rl (
cua_bench.workersstill imports, with a warning). RemoteDesktopSessionkeeps 0.2.7 semantics for code written against it (for example Agents' Last Exam):run_commandreportsreturn_code0 unlessstrict_exit_codes=True, there is no shell timeout unlesstimeout=is set,api_urlsessions speak to computer-server (or cua-spacesd: port 3211 or a sandbox ref such aslocal:<name>,direct:<host:port>), andsession.interfacekeeps the cua-computer method set.scripts/ale_compat/check_ale.pychecks ALE against a checkout.
Running Tests
uv sync --extra dev
uv run pytest -q cua_bench/tests --ignore=cua_bench/tests/live
The suite is hermetic: cua-sandbox is replaced by an in-memory sandbox
(cua_bench/tests/fakes.py) and a loopback computer-server stand-in, so nothing starts a
container or VM. It includes the CLI parser golden
(tests/golden/cli_tree.json), export schema snapshots and the ALE compatibility tests.
After a deliberate, additive CLI or schema change, regenerate the goldens with
CUA_BENCH_UPDATE_GOLDENS=1 and review the diff.
Opt-in live tests (they start real sandboxes):
CUA_BENCH_E2E_LOCAL=1 uv run pytest -s cua_bench/tests/live/test_local_live.py
The ALE check runs agents-last-exam's own loader and harness against this cua-bench:
python scripts/ale_compat/check_ale.py --ale ../agents-last-exam --level 3
Metadata
Release files for cua-bench 0.4.8
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| cua_bench-0.4.8.tar.gz | 245.1 kB | Without provenance | Details for cua_bench-0.4.8.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| cua_bench-0.4.8-py3-none-any.whl | Python 3 | none | any | Details for cua_bench-0.4.8-py3-none-any.whl |
Total release size: 525.9 kB
Release files / cua_bench-0.4.8.tar.gz
| Download URL | cua_bench-0.4.8.tar.gz |
|---|---|
| Size | 245.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
99b9e70f5a99f02e8be0092a3af2215d87eb8d5790537879600bf1b996906c71
|
|
BLAKE2b-256 checksum How to use checksums |
a4797c0d054c17e1562ddb3bccffb70f12b14bd306c3f8b0083c54eb5b2484ca
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.9
|
Release files / cua_bench-0.4.8-py3-none-any.whl
| Download URL | cua_bench-0.4.8-py3-none-any.whl |
|---|---|
| Size | 280.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
38cbe5f8470e72abaa2d358fe75863f22d31c49e66b193ea48212c296c7242b1
|
|
BLAKE2b-256 checksum How to use checksums |
5350aa662a6143b1ed7bb1eab62b1b1aed9b824b88e3e43901abeb4bdabe999e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.9
|
Security for cua-bench 0.4.8
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.