thetabench 0.2.0

Gymnasium-compatible SDK for ThetaBench web agent training & evaluation

ThetaBench Python SDK

Gymnasium-compatible SDK for training and evaluating web agents on ThetaBench environments.

Installation

pip install thetabench

# With browser-mode support (Playwright)
pip install 'thetabench[browser]'
playwright install chromium

Quick start

The SDK talks to a running ThetaBench server over HTTP. Point it at any URL that exposes the standard /api/health, /api/sim/*, and /api/rl routes — a local dev server, a Vercel deployment, or your own self-hosted instance.

REST mode (fast — 1-5 ms / step, ideal for RL training)

import thetabench

env = thetabench.make("http://localhost:3000", task_id="<task-id>")
obs, info = env.reset()

print(f"Task: {info['task_goal']}")
print(f"Max steps: {info['max_steps']}")

done = False
while not done:
    action = my_agent(obs, info)
    obs, reward, terminated, truncated, info = env.step(action)
    done = terminated or truncated

result = env.finish()
print(f"Score: {result['score']:.0%} | Steps: {result['steps']} | Reward: {result['total_reward']:.2f}")
env.close()

Browser mode (realistic — Chromium + accessibility tree, ~100-500 ms / step)

import thetabench

env = thetabench.make(
    "http://localhost:3000",
    task_id="<task-id>",
    mode="browser",
    headless=True,
)
obs, info = env.reset()

# obs includes screenshot, accessibility tree, URL, and structured state
print(f"URL: {obs['url']}")

obs, reward, terminated, truncated, info = env.step({"type": "navigate", "url": "/some/path"})
obs, reward, terminated, truncated, info = env.step({"type": "fill", "selector": "#price", "value": "34.99"})
obs, reward, terminated, truncated, info = env.step({"type": "click", "selector": "button:has-text('Save')"})

result = env.finish()
env.close()

Curriculum training

import thetabench

def my_agent(obs, info):
    # your decision logic here
    return {"action": "navigate", "url": "/some/path"}

runner = thetabench.CurriculumRunner(
    base_url="http://localhost:3000",
    mastery_threshold=0.8,
)

for stage_result in runner.run(my_agent):
    print(f"Stage {stage_result['stage']}: {stage_result['title']}")
    print(f"  Score: {stage_result['avg_score']:.0%}")
    print(f"  Mastery: {'Yes' if stage_result['mastery_achieved'] else 'No'}")

runner.close()

Browse the task catalog

from thetabench import ThetaBenchClient

client = ThetaBenchClient("http://localhost:3000")

# List all tasks on the connected server
tasks = client.list_tasks()
print(f"Total tasks: {tasks.total}")

# Filter by domain / difficulty / type / curriculum stage
filtered = client.list_tasks(domain="<domain>", difficulty="easy")
for t in filtered.tasks:
    print(f"  {t.id}: {t.title}")

# Walk the curriculum
curriculum = client.get_curriculum()
for stage in curriculum.stages:
    print(f"Stage {stage.stage}: {stage.title} ({len(stage.taskIds)} tasks)")

# Server self-identity (site, version, task counts)
health = client.fetch_health()
print(f"Connected to: {health['site']} (v{health['version']}, {health['tasks']} tasks)")

client.close()

CLI

thetabench --url http://localhost:3000 info
thetabench --url http://localhost:3000 tasks --domain <domain> --difficulty easy
thetabench --url http://localhost:3000 run --task <task-id> --agent path/to/your_agent.py
thetabench --url http://localhost:3000 eval --agent path/to/your_agent.py --output results.json

Your agent file must export agent_step(obs, info) -> action. Optionally agent_response(obs, info) -> str | None for retrieval / impossibility tasks. Optionally a module-level MODEL = "..." for results metadata.

API Reference

thetabench.make(base_url, task_id, *, mode='rest', **kwargs) -> ThetaBenchEnv

Factory that constructs a ThetaBenchEnv pointed at a running server.

Param	Type	Default	Description
base_url	str	(required)	Server URL — local dev server, Vercel deployment, or self-hosted.
task_id	str	(required)	Task identifier exposed by the server's /api/sim/tasks.
mode	"rest" | "browser"	"rest"	Transport mode.
**kwargs			Forwarded to ThetaBenchEnv — seed, config_overrides, headless, viewport, render_mode.

The server self-identifies its site at /api/health; the SDK does not maintain a hardcoded site → URL map.

ThetaBenchEnv (Gymnasium)

Method	Returns	Description
reset()	(obs, info)	Start a new episode.
step(action)	(obs, reward, terminated, truncated, info)	Execute one action.
evaluate()	dict	Mid-episode score check (non-destructive).
finish(agent_response=None)	dict	End the episode and get the final score.
close()	None	Release HTTP + browser resources.

CurriculumRunner

Method	Description
run(agent_fn, agent_response_fn=None, start_stage=1)	Generator yielding stage results, advancing on mastery.
run_stage(stage, agent_fn, ...)	Run all tasks in one stage.
run_task(task_id, agent_fn, ...)	Run a single task.
get_curriculum()	List of stages from the server.

BatchRunner

Method	Description
run_all(domain=None, difficulty=None, task_type=None)	Run every (filtered) task; returns standardized JSON results with per-domain and per-stage breakdowns. The meta.site field is filled from the server's /api/health response.
close()	Close the underlying HTTP client.

ThetaBenchClient

Thin typed httpx wrapper over the standard endpoints. See the source for the full method list — start_episode, finish_episode, evaluate, step, observe, reset_env, get_state, get_snapshot, get_episode, list_tasks, get_task, get_curriculum, log_action, get_action_space, fetch_health.

License

MIT — see LICENSE.