cua-bench 0.2.0

Platform for creating computer-use verifiable environments and training VLM agents to use them.

cua-bench

A set of tools for creating verifiable environments for computer automation tasks, evaluation, and training. Features support for both real Windows, Linux, macOS, and Android VM environments, as well as HTML-based webtop environments that can visually emulate macos, win11, win10, ios, android, and more.

Installation

uv tool install -e .
playwright install chromium

Docker Setup (for batch jobs and dataset processing)

Build the cua-bench Docker image:

docker build -t cua-bench:latest .

Quick Start

Create an environment

cb create-task tasks/my_env

Run the environment:

cb interact tasks/my_env

CLI Usage

Install an environment

cb install tasks/click_env

List tasks

# List all environments
cb tasks

# List tasks in specific environment
cb tasks tasks/click_env

Interact with a task

Interact with a task in the browser. This is useful for debugging and testing.

cb interact tasks/click_env --task-id 0 --solve --screenshot output.png

Evaluate agents on tasks

# Evaluate agent on tasks/click_env
cb eval tasks/click_env --model anthropic/claude-3-5-sonnet-20240620

Run tasks with batch processing

Run a cluster of cua-bench tasks on GCP or locally. For multi-step trajectories, use cb dump-solution. For single-step trajectories, use cb dump-setup.

# Build Docker image first (required for local batch)
docker build -t cua-bench:latest .

# Local (Docker) - Run 4 tasks from click_env (setup + solve + evaluate)
cb dump-solution tasks/click_env 4 --local

# Local (Docker) - Run 4 tasks from click_env (setup + evaluate)
cb dump-setup tasks/click_env 4 --local --output-dir ./outputs

# GCP Batch - Run 16 tasks from click_env (setup + solve + evaluate)
cb dump-solution tasks/click_env 16 --parallelism 8

# GCP Batch - Run 16 tasks from click_env (setup + evaluate)
cb dump-setup tasks/click_env 16 --parallelism 8 --output-dir ./outputs

Process snapshots into a training dataset for UI grounding

Given a directory of snapshots, cua-bench offers a simple way to process them into a dataset for UI grounding using action augmentation.

# Process 5 snapshots using 'aguvis' action augmentation
cb process ./outputs 5

# Process all snapshots and push to Hugging Face Hub
cb process ./outputs --push-to-hub --repo-id username/repo

Programmatic Interface

import cua_bench as cb

# Create an environment
env = cb.make("tasks/click_env")

# Setup and get initial screenshot
screenshot, task_cfg = env.reset()  # optionally pass task_id

# Execute a step
screenshot = env.step('page.click("#submit")')

# Run the solution
screenshot = env.solve()

# Evaluate the result
rewards = env.evaluate()

# Clean up
env.close()