scalewob 0.10.2

Python SDK for evaluating in ScaleWoB: Scalable world-of-bit that revolutionizes the evaluation of Computer-Use Agents.

ScaleWoB Python SDK

Python SDK for evaluating in ScaleWoB: Scalable world-of-bit that revolutionizes the evaluation of Computer-Use Agents.

🔥 Use this SDK to plug your computer-use agent to our upcoming benchmark!

Installation

pip install scalewob

Discovering Tasks

ScaleWoB uses a two-step process to discover tasks:

1. Fetch available environments
2. Load each environment to get its tasks

from scalewob import ScaleWoBAutomation, fetch_environments

# Step 1: Fetch available environments
envs = fetch_environments()
print(f"Found {len(envs)} environments")

# Filter by difficulty
basic_envs = fetch_environments(difficulty="Basic")

# Filter by platform and tags
mobile_envs = fetch_environments(
    platform="Mobile Interfaces",
    tags=["Time Selection"]
)

# Step 2: For each environment, load it and work with its tasks
for env in basic_envs:
    with ScaleWoBAutomation(env_id=env['id']) as auto:  # the browser will show up
        # Access tasks via the tasks property
        print(f"Environment {env['name']} has {len(auto.tasks)} tasks")

        # Work with each task
        for idx, task in enumerate(auto.tasks):
            auto.start_evaluation()
            # ... perform actions based on task['description'] ...

            # The task's params field (if present) is a JSON schema
            # You need to provide actual parameter values that match this schema
            if task.get('params'):
                # Example: provide actual values matching the schema
                # Validation is automatic when you call finish_evaluation
                actual_params = {
                    "destination": "New York",
                    "check_in": "2024-01-15"
                }
                result = auto.finish_evaluation(
                    task_id=task['task_id'],
                    params=actual_params  # Auto-validated against task['params']
                )
            else:
                result = auto.finish_evaluation(task_id=task['task_id'])

Development

Setup

# Clone the repo and enter the directory first
uv sync

# Install pre-commit hooks
uv pre-commit install

Code Quality

# Format code
uv run poe format

# Run checks (format, lint, type checking)
uv run poe check

# Fix linting issues
uv run poe fix

License

MIT License - see LICENSE file for details.

Links

• Homepage: https://github.com/ScaleWoB/ScaleWoB.github.io
• Documentation: https://github.com/ScaleWoB/ScaleWoB#readme
• Bug Tracker: https://github.com/ScaleWoB/ScaleWoB/issues