scalewob 0.8.0

Python SDK for evaluating in ScaleWoB: Scalable world-of-bit that revolutionizes the evaluation of Computer-Use Agents.

ScaleWoB Python SDK

Python SDK for evaluating in ScaleWoB: Scalable world-of-bit that revolutionizes the evaluation of Computer-Use Agents.

🔥 Use this SDK to plug your computer-use agent to our upcoming benchmark!

Installation

pip install scalewob

Discovering Tasks

ScaleWoB uses a two-step process to discover tasks:

1. Fetch available environments
2. Load each environment to get its tasks

from scalewob import ScaleWoBAutomation, fetch_environments

# Step 1: Fetch available environments
envs = fetch_environments()
print(f"Found {len(envs)} environments")

# Filter by difficulty
basic_envs = fetch_environments(difficulty="Basic")

# Filter by platform and tags
mobile_envs = fetch_environments(
    platform="Mobile Interfaces",
    tags=["Time Selection"]
)

# Step 2: For each environment, load it and work with its tasks
for env in basic_envs:
    auto = ScaleWoBAutomation(env_id=env['id'])
    auto.start()  # Tasks are automatically fetched here

    # Access tasks via the tasks property
    print(f"Environment {env['name']} has {len(auto.tasks)} tasks")

    # Work with each task
    for idx, task in enumerate(auto.tasks):
        auto.start_evaluation()
        # ... perform actions based on task['description'] ...

        # The task's params field (if present) is a JSON schema
        # You need to provide actual parameter values that match this schema
        if task.get('params'):
            # Example: provide actual values matching the schema
            # Validation is automatic when you call finish_evaluation
            actual_params = {
                "destination": "New York",
                "check_in": "2024-01-15"
            }
            result = auto.finish_evaluation(
                task_id=task['task_id'],
                params=actual_params  # Auto-validated against task['params']
            )
        else:
            result = auto.finish_evaluation(task_id=task['task_id'])

    auto.close()

See Environment Discovery and ScaleWoBAutomation Methods in the API Reference for more details.

Development

Setup

# Clone the repo and enter the directory first
uv sync

# Install pre-commit hooks
uv pre-commit install

Code Quality

# Format code
uv run poe format

# Run checks (format, lint, type checking)
uv run poe check

# Fix linting issues
uv run poe fix

License

MIT License - see LICENSE file for details.

Links

• Homepage: https://github.com/ScaleWoB/ScaleWoB.github.io
• Documentation: https://github.com/ScaleWoB/ScaleWoB#readme
• Bug Tracker: https://github.com/ScaleWoB/ScaleWoB/issues