scalewob 0.5.0

Python SDK for evaluating in ScaleWoB: Scalable world-of-bit that revolutionizes the evaluation of Computer-Use Agents.

ScaleWoB Python SDK

Python SDK for evaluating in ScaleWoB: Scalable world-of-bit that revolutionizes the evaluation of Computer-Use Agents.

🔥 Use this SDK to plug your computer-use agent to our upcoming benchmark!

Installation

pip install scalewob

Quick Start

from scalewob import ScaleWoBAutomation

# Initialize automation for a specific environment
auto = ScaleWoBAutomation(env_id='booking-hotel-simple')

# Start browser and load environment
auto.start()

# Start evaluation mode
auto.start_evaluation()

# Perform actions using coordinates
auto.click(x=300, y=150)  # Click at coordinates
auto.type('New York')      # Type into focused element

# Finish evaluation and get results
result = auto.finish_evaluation({'destination': 'New York'})
print(result)

# Clean up
auto.close()

Discovering Environments

Fetch available environment metadata from the ScaleWoB registry:

from scalewob import fetch_environments

# Get all available environments
envs = fetch_environments()
print(f"Found {len(envs)} environments")

# Filter by difficulty
expert_envs = fetch_environments(difficulty="Expert")

# Filter by platform and tags
time_selection_envs = fetch_environments(
    platform="Mobile Interfaces",
    tags=["Time Selection"]
)

See Environment Discovery in the API Reference for more details.

Usage

Context Manager

with ScaleWoBAutomation(env_id='booking-hotel-simple') as auto:
    auto.start()
    auto.start_evaluation()
    auto.click(x=300, y=150)
    auto.type('New York')
    result = auto.finish_evaluation({'destination': 'New York'})

Configuration

auto = ScaleWoBAutomation(
    env_id='booking-hotel-simple',
    headless=False,             # Run in headless mode
    base_url='https://niumascript.com/scalewob-env',
    timeout=5000,               # Default timeout in milliseconds
    screenshot_quality='high'   # 'low' (1x) or 'high' (3x) scale
)

API Reference

Initialization

ScaleWoBAutomation(env_id, headless=False, base_url='https://niumascript.com/scalewob-env', timeout=5000, screenshot_quality='high')

Initialize automation interface for ScaleWoB environments.

Parameters:

• env_id (str): Environment ID to launch
• headless (bool): Run browser in headless mode (default: False). Uses Chrome browser.
• base_url (str): Base URL for ScaleWoB environments (default: 'https://niumascript.com/scalewob-env')
• timeout (int): Default timeout for operations in milliseconds (default: 5000)
• screenshot_quality (str): Screenshot quality - 'low' for 1x scale, 'high' for 3x scale (default: 'high')

Note: Currently only Chrome browser is supported. The browser runs with stealth mode options to avoid detection.

Core Methods

start()

Initialize Chrome browser and navigate to the environment page. Must be called before any other automation methods. Waits for DOM to be fully loaded before returning.

start_evaluation()

Start evaluation mode. Ensures the environment is fully initialized and clears the trajectory for a fresh evaluation. The environment loads ready to interact without requiring UI button clicks.

finish_evaluation(params=None)

Finish evaluation and get results.

Parameters:

• params (dict, optional): Evaluation parameters (environment-specific)

Returns: Evaluation result dictionary

Interaction Methods

click(x, y)

Click at coordinates (x, y).

Parameters:

• x (int): Horizontal coordinate
• y (int): Vertical coordinate

type(text, append=False)

Type text into the currently focused element. An element must be focused first (e.g., via click).

Parameters:

• text (str): Text to type
• append (bool): If True, append to existing text; if False, clear field first (default: False)

scroll(x, y, direction='down', distance=100)

Scroll in direction from coordinates (x, y).

Parameters:

• x (int): Horizontal coordinate
• y (int): Vertical coordinate
• direction (str): Scroll direction ('up', 'down', 'left', 'right')
• distance (int): Distance to scroll in pixels

long_press(x, y, duration=1000)

Long press at coordinates (x, y).

Parameters:

• x (int): Horizontal coordinate
• y (int): Vertical coordinate
• duration (int): Duration of press in milliseconds

drag(x, y, end_x, end_y)

Drag from start coordinates to end coordinates.

Parameters:

• x (int): Starting horizontal coordinate
• y (int): Starting vertical coordinate
• end_x (int): Ending horizontal coordinate
• end_y (int): Ending vertical coordinate

back()

Go back in navigation history.

State and Information Methods

take_screenshot(format='base64')

Capture screenshot of the environment.

Parameters:

• format (str): Return format - "base64" for raw base64 string, "pil" for PIL Image object

Returns: Base64 string or PIL Image object

get_evaluation_result()

Get the last evaluation result.

Returns: Last evaluation result or None

get_trajectory()

Get current action trajectory.

Returns a copy of the trajectory history containing all actions performed since start_evaluation() was called.

Returns: List of trajectory entries with timestamp, type, and data

Example:

trajectory = auto.get_trajectory()
print(f"Collected {len(trajectory)} actions")
for action in trajectory:
    print(f"{action['type']} at {action['timestamp']}")

clear_trajectory()

Clear the current trajectory history.

This is useful if you want to reset the trajectory without restarting the evaluation. Note that start_evaluation() automatically clears the trajectory.

Example:

auto.clear_trajectory()
print(len(auto.get_trajectory()))  # 0

close()

Close browser and cleanup resources.

Environment Discovery

fetch_environments(difficulty=None, platform=None, tags=None, force_refresh=False)

Fetch environment metadata from ScaleWoB registry with optional filtering.

Parameters:

• difficulty (str, optional): Filter by difficulty level (e.g., "Basic", "Advanced", "Expert")
• platform (str, optional): Filter by platform (e.g., "Mobile Interfaces")
• tags (list, optional): Filter by tags (returns environments matching any tag)
• force_refresh (bool): Bypass cache and fetch fresh data (default: False)

Returns: List of environment metadata dictionaries

Raises: NetworkError if fetching or parsing fails

Example:

from scalewob import fetch_environments

# Get all environments
all_envs = fetch_environments()

# Filter by multiple criteria
filtered = fetch_environments(
    difficulty="Expert",
    platform="Mobile Interfaces"
)

# Force refresh cache
fresh = fetch_environments(force_refresh=True)

Exception Handling

from scalewob import (
    ScaleWoBError,      # Base exception
    TimeoutError,       # Operation timeout
    CommandError,       # Command execution failure
    EvaluationError,    # Evaluation failure
    BrowserError,       # Browser automation failure
    NetworkError        # Network operation failure
)

try:
    auto = ScaleWoBAutomation(env_id='booking-hotel-simple')
    auto.start()
    auto.start_evaluation()
    result = auto.finish_evaluation()
except TimeoutError as e:
    print(f"Operation timed out: {e}")
except EvaluationError as e:
    print(f"Evaluation failed: {e}")
except ScaleWoBError as e:
    print(f"ScaleWoB error: {e}")
finally:
    auto.close()

Development

Setup

# Clone the repo and enter the directory first
uv sync

# Install pre-commit hooks
uv pre-commit install

Code Quality

# Format code
uv run poe format

# Run checks (format, lint, type checking)
uv run poe check

# Fix linting issues
uv run poe fix

License

MIT License - see LICENSE file for details.

Links

• Homepage: https://github.com/ScaleWoB/ScaleWoB.github.io
• Documentation: https://github.com/ScaleWoB/ScaleWoB#readme
• Bug Tracker: https://github.com/ScaleWoB/ScaleWoB/issues