cua-bench 0.2.11

Toolkit for computer-use RL environments and benchmarks

cua-bench

Framework for benchmarking Computer-Use Agents with verifiable cross-platform environments.

Documentation - Installation, guides, and API reference.

Running Tests

The test suite covers the core gym interface, worker system, and benchmark runners.

Install dev dependencies

uv pip install -e ".[dev,browser,server,rl]"

Note: The browser extra installs Playwright for e2e tests with the simulated provider.

Run all tests

uv run --with pytest pytest cua_bench/tests/ -v

Run specific test modules

# Core gym interface (make, reset, step, evaluate)
uv run --with pytest pytest cua_bench/tests/test_gym_interface.py -v

# HTTP worker client (/reset, /step endpoints)
uv run --with pytest pytest cua_bench/tests/test_worker_client.py -v

# Worker server endpoints and action serialization
uv run --with pytest pytest cua_bench/tests/test_worker_server.py -v

# Benchmark runner functions
uv run --with pytest pytest cua_bench/tests/test_run_benchmark.py -v

# Worker manager (spawning/managing workers)
uv run --with pytest pytest cua_bench/tests/test_worker_manager.py -v

# Action parsing
uv run --with pytest pytest cua_bench/tests/test_actions.py -v

Run tests with coverage

uv run --with pytest --with pytest-cov pytest cua_bench/tests/ -v --cov=cua_bench --cov-report=term-missing

Test Structure

Test Module	What it Tests	Approach
test_gym_interface.py	Core Environment API: make(), reset(), step(), evaluate()	E2E - Real simulated (Playwright) environments
test_worker_client.py	HTTP client for worker servers (CBEnvWorkerClient)	Mock server - Uses @patch("requests.post") to mock HTTP responses
test_worker_server.py	FastAPI endpoints and action serialization	Unit - Action serialize/deserialize, request models, simple endpoints
test_run_benchmark.py	run_benchmark(), run_single_task(), run_interactive()	E2E - Real simulated (Playwright) environments
test_worker_manager.py	Workers + dataloader training loop	E2E - Real workers, real envs, mock model for actions
test_actions.py	Action string parsing (repr_to_action())	Unit - Pure function tests

Test Approach Philosophy

• E2E tests use real simulated (Playwright) environments. The simulated provider is fast enough for testing.
• Mock server tests (test_worker_client.py) mock HTTP responses to test client logic in isolation.
• Mock model (test_worker_manager.py) uses a mock model that returns simple actions to test the dataloader training loop without requiring a real ML model.

Infrastructure Benchmarking

Measure the throughput of the worker infrastructure:

uv run python -m cua_bench.scripts.benchmark_workers --num_workers 16 --num_steps 10

Options:

Flag	Default	Description
--num_workers	16	Number of parallel workers
--num_steps	10	Steps per worker
--task_path	None	Path to task directory (creates temp if empty)

Output:

• Average reset time
• Average step time
• Average finish time
• Step throughput (steps/sec)