GL Computer Use

Description

A typed Python SDK for desktop automation via natural-language prompts. GL Computer Use wraps cloud desktop sandboxes and computer-use agents into a clean async API with live streaming, human-in-the-loop takeover, structured observability, and swappable providers.

Key Features

• Streaming and non-streaming run modes: run() for live events, run_once() for a single result, run_sync() for non-async scripts and Jupyter notebooks.
• agent="openai" (default): drives OpenAI's Responses API "computer" tool directly (the current GA tool — not the retired computer_use_preview) — see gl_computer_use/agent/openai_agent.py. Added alongside Agent-S after cua was removed, to keep a direct OpenAI computer-use option available without a per-provider routing layer to fall behind new models. Requires GLCU_OPENAI_API_KEY and a model that supports the computer tool (default openai/gpt-5.6-sol). The only agent that implements supports_trajectory_persistence (resume seeds previous_response_id, subject to OpenAI's response-retention window). Verified end-to-end against a real E2B sandbox + OpenAI account with cost tracking (5 steps, ~$0.017).
• Agent-S (simular-ai) built in: also available (agent="agents"), model-agnostic — works with any provider/model including Anthropic's. Register a custom agent via register_agent() to use a different backend.
• Swappable execution environments: e2b (E2B Desktop, default), opensandbox (Alibaba OpenSandbox), or local for explicit, unsandboxed control of the active host desktop.
• Live desktop URL: noVNC streaming URL surfaced via the SANDBOX_READY event or StreamClient.stream_url.
• Human-in-the-loop takeover: pause an agent loop and hand control to a human, then resume with optional guidance.
• Artifact storage: local disk by default, MinIO/S3 via the minio extra.
• Structured logging with optional OpenTelemetry tracing/metrics and Sentry via the observability extra.
• Custom provider registration: plug in your own sandbox, agent, or artifact store without modifying the SDK.

---

Installation

Install the core SDK:

pip install gl-computer-use

This includes both built-in agents: the default agent="openai" (just the openai package, lightweight) and Agent-S (agent="agents"), since gui-agents is a required dependency regardless of which agent you actually use (note: gui-agents pulls in a fairly heavy transitive dependency tree, including PaddleOCR/PaddlePaddle).

Install optional extras only when you need them:

pip install "gl-computer-use[recording]"     # WebM session recording via Playwright
pip install "gl-computer-use[opensandbox]"   # Alibaba OpenSandbox support
pip install "gl-computer-use[minio]"         # MinIO / S3-compatible artifact store
pip install "gl-computer-use[observability]" # OTLP tracing/metrics + Sentry via gl-observability
pip install "gl-computer-use[all]"           # all of the above

API keys required at runtime:

1. E2B API key — e2b.dev (when using sandbox="e2b")
2. OpenAI API key (for the default agent="openai" / gpt-5.6-sol model) or Anthropic API key (when using agent="agents" with an anthropic/* model)

Session recording setup (optional, one-time)

WebM recordings require Playwright's Chromium binaries (~130 MB, stored under ~/.cache/ms-playwright/):

pip install "gl-computer-use[recording]"
gl-computer-use-setup

If you skip this step, the SDK falls back to GIF recording via screenshot stitching.

---

Quick Start

Streaming events

run() returns a StreamClient; iterate it to receive events. The terminal TASK_COMPLETED event carries the final TaskResult.

import asyncio
from gl_computer_use import GLComputerUseClient


async def main() -> None:
    client = GLComputerUseClient()
    stream = await client.run("Open Firefox and navigate to google.com")

    async for event in stream:
        if event.event_type == "SANDBOX_READY" and event.stream_url:
            print(f"Watch live at: {event.stream_url}")
        elif event.event_type == "STEP_COMPLETED":
            print(f"Step {event.step_index}: {event.action.type if event.action else '—'}")
        elif event.event_type == "TASK_COMPLETED":
            print(f"Status: {event.result.status}")
            print(f"Output: {event.result.output}")


asyncio.run(main())

Fire-and-forget async

run_once() returns a TaskResult directly when the task finishes. Raises TaskFailedError / TaskCancelledError on non-COMPLETED outcomes.

import asyncio
from gl_computer_use import GLComputerUseClient


async def main() -> None:
    client = GLComputerUseClient()
    result = await client.run_once("Open a terminal and check Python version")
    print(result.status, result.output, len(result.steps))


asyncio.run(main())

Synchronous / Jupyter

run_sync() is a plain synchronous method — no asyncio.run(), no await. It detects whether an event loop is already running and dispatches via ThreadPoolExecutor when needed, so it works in regular scripts and Jupyter notebooks (no nest_asyncio required).

from gl_computer_use import GLComputerUseClient

result = GLComputerUseClient().run_sync("Open the file manager")
print(result.status)

---

Configuration

Configuration is read from environment variables (prefix GLCU_) or by passing a GLComputerUseConfig object directly. Create a .env file in your working directory:

GLCU_E2B_API_KEY=sk-e2b-...
GLCU_OPENAI_API_KEY=sk-...

# Optional overrides
GLCU_MODEL=openai/gpt-5.6-sol
GLCU_TASK_TIMEOUT=300
GLCU_MAX_STEPS=50

Critical fields:

Variable	Default	Description
GLCU_E2B_API_KEY	None	E2B Desktop API key (required when sandbox="e2b")
GLCU_OPENAI_API_KEY	None	OpenAI API key (required for the default agent="openai" and for openai/* models)
GLCU_ANTHROPIC_API_KEY	None	Anthropic API key (required for anthropic/* models, e.g. with agent="agents")
GLCU_MODEL	"openai/gpt-5.6-sol"	LLM in provider/name format; must support the computer tool when agent="openai"
GLCU_AGENT	"openai"	Agent provider — "openai" (direct OpenAI computer-use tool) or "agents" (Agent-S, model-agnostic); pass a custom name registered via register_agent() to use a different backend
GLCU_SANDBOX	"e2b"	Execution provider: "e2b", "opensandbox", or "local"; local directly controls the active host desktop without isolation
GLCU_ARTIFACT	"local"	Artifact store: "local" or "minio"
GLCU_TASK_TIMEOUT	300.0	How long one run may take, in seconds
GLCU_SANDBOX_TIMEOUT	300	Sandbox lifetime in seconds, measured from create (E2B clamps to 3600) — equal to GLCU_TASK_TIMEOUT by default (a deliberate shipped default; a task only loses its tail when it runs nearly the full budget); raise it above GLCU_TASK_TIMEOUT for headroom if provisioning and bring-up are slow on your backend
GLCU_SANDBOX_IMAGE	""	Image/template to boot; empty means the backend default (desktop for E2B, public.ecr.aws/c1z4u5m0/external/desktop-sandbox:latest for OpenSandbox)
GLCU_SANDBOX_PROVISION_TIMEOUT	300	HTTP timeout for create/resume calls, and the budget for polling a snapshot to Ready — raise it when the server is slow to answer
GLCU_SANDBOX_READY_TIMEOUT	30	OpenSandbox health-check timeout after create/restore; an unready sandbox is killed when this expires — raise it when cold image pulls are slow
GLCU_SANDBOX_REQUEST_TIMEOUT	60	HTTP timeout for actions and screenshots once the sandbox is up
GLCU_MAX_STEPS	50	Maximum agent loop iterations; also the Agent-S ceiling unless GLCU_AGENTS_MAX_STEPS is set
GLCU_OPENAI_REASONING_EFFORT	unset	OpenAI reasoning effort for agent="openai": none, minimal, low, medium, high, xhigh
GLCU_AGENTS_TEMPERATURE	None	Sampling temperature override for Agent-S LLM calls; unset leaves gui-agents' own default (0.0), which some newer reasoning-tier OpenAI models reject outright
GLCU_LOCAL_ARTIFACT_DIR	"./artifacts"	Directory for saved screenshots and recordings
GLCU_LOG_LEVEL	"INFO"	DEBUG, INFO, WARNING, or ERROR
GLCU_LOG_FORMAT	"json"	"json" (structured) or "console" (human-readable)
GLCU_DEFAULT_DISPOSITION	"destroy"	Teardown when no disposition is passed: "destroy" or "snapshot" (see Snapshot & Resume)
GLCU_STRICT_SNAPSHOT	False	Turn best-effort snapshot/resume failures into hard ConfigError/SnapshotError
GLCU_TRAJECTORY_MAX_IMAGES	None	Reserved for a future agent with image-capped trajectory replay; no built-in agent implements it, so any value other than None raises ConfigError at config time (see Agent support)
GLCU_TRAJECTORY_MAX_BYTES	None	Hard ceiling (bytes) on persisted trajectory JSON
GLCU_TRAJECTORY_PII_ANONYMIZATION	False	Reserved for a future agent with trajectory PII anonymization; no built-in agent implements it, so True raises ConfigError at config time (see Agent support) — not to be confused with GLCU_PII_REDACTION_ENABLED, which still works and redacts PII from log lines, not trajectories
GLCU_KEEP_SNAPSHOT_HISTORY	False	Retain every snapshot instead of rolling-GC'ing predecessors (E2B)
GLCU_ALLOW_MODEL_DRIFT	False	Allow resuming a token whose model differs from the configured model

OpenSandbox, MinIO, Agent-S, and observability (OTLP/Sentry/PII) have additional GLCU_* env vars — see GLComputerUseConfig in gl_computer_use/config.py for the full list.

Timeouts

The five timeout knobs nest, and three of them are named *_SANDBOX_*, so it is easy to reach for the wrong one:

Knob	Bounds	Raise it when
GLCU_SANDBOX_TIMEOUT	Total sandbox lifetime (wall clock, from create)	Long tasks die mid-run as transport errors
GLCU_TASK_TIMEOUT	One run (wall clock)	The agent legitimately needs more steps
GLCU_SANDBOX_PROVISION_TIMEOUT	A single HTTP request while creating/resuming; also the snapshot-to-Ready polling budget	The server is slow to answer
GLCU_SANDBOX_READY_TIMEOUT	The OpenSandbox health check, after which the SDK kills the sandbox	Cold image pulls are slow
GLCU_SANDBOX_REQUEST_TIMEOUT	A single HTTP request once the desktop is up	Actions or screenshots time out on a healthy box

PROVISION_TIMEOUT and REQUEST_TIMEOUT bound one request each, not a phase — a phase issuing twenty calls can far outlast either. Only SANDBOX_TIMEOUT and TASK_TIMEOUT bound elapsed time.

READY_TIMEOUT is the one to size against your own server: run a task on an uncached node once and check how long provisioning takes before the health check passes.

---

Provider Agnosticism

Swap sandboxes (and custom-registered agents) via config alone — no code changes:

Agent	Sandbox	Config
OpenAI (default)	E2B (default)	GLComputerUseClient()
OpenAI	OpenSandbox	GLComputerUseConfig(sandbox="opensandbox")
OpenAI	Local host desktop	GLComputerUseConfig(sandbox="local")
Agent-S	E2B	GLComputerUseConfig(agent="agents", model="anthropic/claude-sonnet-4-6")
Agent-S	OpenSandbox	GLComputerUseConfig(agent="agents", sandbox="opensandbox", model="anthropic/claude-sonnet-4-6")

from gl_computer_use import GLComputerUseClient, GLComputerUseConfig

client = GLComputerUseClient(GLComputerUseConfig(sandbox="opensandbox"))  # default agent="openai"

# Or swap to Agent-S with a different model provider:
client = GLComputerUseClient(
    GLComputerUseConfig(agent="agents", sandbox="opensandbox", model="anthropic/claude-sonnet-4-6")
)

Direct local desktop execution

sandbox="local" runs computer actions against the active desktop of the machine running Python. It does not create a container or VM, and cleanup never shuts down or logs out the host. Keep the desktop unlocked and visible throughout the run.

from gl_computer_use import GLComputerUseClient, GLComputerUseConfig

config = GLComputerUseConfig(
    sandbox="local",
    agent="openai",
    model="openai/gpt-5.6-sol",
    openai_api_key="sk-...",
)
result = GLComputerUseClient(config).run_sync("Open the calculator application")
print(result.output)

This mode is intentionally explicit because it has no isolation: the agent can see the screen, operate applications, and affect real accounts and files. Use a dedicated OS account, close sensitive applications, and keep PyAutoGUI's corner fail-safe enabled. macOS requires Screen Recording and Accessibility permissions; Linux requires an X11 session for PyAutoGUI input. On Ubuntu, install PyAutoGUI's screenshot dependency before running locally:

sudo apt install scrot

On macOS, provision() preflights the Screen Recording grant and fails with an actionable error when it is missing, because a denied grant makes every screenshot blank without raising anything.

Only one local run may be active on a host at a time — two agents sharing one keyboard and pointer would corrupt each other's runs. The claim is held in a lock file under the system temporary directory and is released by the operating system even if the owning process crashes, so a second run provisioned while the first is still alive fails immediately with the owner's process id.

Local runs have no VNC stream, so there is no page for Playwright to record. The runner falls back to stitching the per-step screenshots into a GIF, which is the recording artifact a local run produces.

---

Runtime API

The client exposes three run methods:

Method	Returns	Use when
await client.run(prompt, ...)	StreamClient	You need live event streaming or the SANDBOX_READY URL before the task finishes
await client.run_once(prompt, ...)	TaskResult	You only need the final result, async context
client.run_sync(prompt, ...)	TaskResult	You only need the final result, non-async script or Jupyter notebook

All three methods accept the same parameters:

Parameter	Type	Default	Description
prompt	str	—	Task description
config	GLComputerUseConfig | None	None	Per-call config override
timeout	float | None	None	Max seconds (falls back to config.timeout)
files	list[File] | None	None	Files to upload to the sandbox before the task
retrieve_files	list[str] | None	None	Sandbox paths to download after completion
on_takeover_needed	Callable | None	None	Takeover callback

run_once() and run_sync() raise TaskFailedError / TaskCancelledError directly instead of returning a result with a non-COMPLETED status.

---

Live Desktop (noVNC)

When using the E2B sandbox, a noVNC HTTP endpoint is started alongside the desktop. The SDK waits until that endpoint is reachable before surfacing the URL.

# Option A — pre-iteration attribute
stream = await client.run("do something")
print(stream.stream_url)

# Option B — first SANDBOX_READY event
async for event in stream:
    if event.event_type == "SANDBOX_READY" and event.stream_url:
        webbrowser.open(event.stream_url)

---

Takeover

Pass on_takeover_needed to run() / run_once() / run_sync(). The agent pauses when a takeover condition is detected, and your callback receives a TakeoverContext with the session state; return TakeoverResponse(resume=True, message=...) to continue (with optional guidance for the agent) or TakeoverResponse(resume=False) to cancel the task outright.

When a callback is supplied, TakeoverRequiredError is raised directly out of the event stream (async for event in stream), rather than surfaced as a terminal event, in two cases: the callback returned resume=False, or the callback itself raised. In the second case, the original exception is chained as __cause__. The sandbox is torn down before the error propagates. Wrap your loop in try/except TakeoverRequiredError to handle a deliberate cancellation cleanly, and check exc.__cause__ to tell a decline apart from a callback bug. See examples/takeover.py (which does this) and examples/takeover_caller_initiated.py.

With no callback, behavior depends on the entry point:

• run() yields the TAKEOVER_STARTED event to your loop, and the agent stays paused. Resume it yourself with await client.takeover_resume(task_id, message=...). Otherwise no further events arrive, and the stream raises GLTimeoutError (tearing down the sandbox) once timeout elapses. No TakeoverRequiredError is raised.
• run_once() / run_sync() auto-resume by default (auto_resume_on_takeover=True). With auto_resume_on_takeover=False, they behave like run() above: the paused agent ends in GLTimeoutError.

---

Sensitive Data

Pass sensitive_data to give an agent a credential to type without the raw value ever reaching the LLM:

config = GLComputerUseConfig(sensitive_data={"x_username": "alice", "x_password": "hunter2"})
client = GLComputerUseClient(config=config)

async for event in await client.run("Log in with <secret>x_username</secret> / <secret>x_password</secret>"):
    ...

The model is told only the placeholder key names (x_username, x_password), never their values. When the model types <secret>x_password</secret> — or, as a fallback for when it drops the tags, the bare exact key name x_password on its own — the SDK swaps in the real value right before the keystrokes reach the sandbox. This mirrors gl-browser-use's sensitive_data field of the same name and shape.

This is a real gap, not a nice-to-have caveat: sensitive_data is flat and global, with no domain/app scoping. A browser page has a URL to scope a secret against; a desktop session has no equivalent boundary (a window title was considered and rejected — a malicious app could set its own title). Any configured secret is usable by any type action for the whole task, regardless of which window or app it lands in. Do not combine sensitive_data with a task that also has the agent interact with untrusted or attacker-controlled content (e.g. an arbitrary webpage), since nothing stops a prompt-injected instruction from directing a configured secret into it.

Neither the tagged nor the bare-fallback form is foolproof — a model that paraphrases instead of using one of the two recognized forms (e.g. describing the password in prose) won't trigger substitution, and there's no detection for that failure mode today. Validate empirically against your chosen model/agent before relying on this for a real credential.

TOTP/MFA live code generation is not handled by sensitive_data — tracked separately, see GDP-ADMIN/gl-sdk#6867.

sensitive_data is excluded from GLComputerUseConfig's repr() and model_dump(), so it won't leak into logs or config serialization.

---

Snapshot & Resume

A session can be paused — its sandbox state, and the agent's conversation trajectory where the agent supports it (see Agent support below) — and later resumed from a ResumeToken. This is fully backward-compatible: the teardown disposition defaults to destroy, so existing callers are unaffected.

Pass disposition="snapshot" to capture a token, then pass it back via resume_from:

client = GLComputerUseClient()

# 1. Run and snapshot instead of destroying the sandbox.
result = await client.run_once("Open Firefox and log into the dashboard", disposition="snapshot")
token = result.resume_token            # a ResumeToken; token.to_json() to persist it

# 2. Later — resume from where it left off.
result = await client.run_once("Now download this month's report", resume_from=token)

resume_from accepts a ResumeToken, its dict, or its JSON-string form. Set GLCU_DEFAULT_DISPOSITION=snapshot to snapshot by default without passing the argument each call.

Artifact store requirement

Snapshot history is written through the artifact store, so the store must support history persistence. The built-in local and minio stores do; a custom store must set the class attribute supports_history = True and implement save_history / load_history / delete_history.

If the store does not support history, the snapshot still captures the desktop but the token is emitted with history_ref=None — the restored sandbox has no agent memory. By default this is logged at ERROR and the run continues. Set strict_snapshot=True (GLCU_STRICT_SNAPSHOT=true) to turn it — and any history-persistence or snapshot-capture failure — into a hard ConfigError / SnapshotError instead.

Agent support

Trajectory capture/replay is opt-in per agent (BaseAgent.supports_trajectory_persistence), not automatic. agent="openai" (the default) implements it — resume seeds previous_response_id from the prior run, so the model retains memory of the conversation, subject to OpenAI's response-retention window (~30 days; a resume attempted after that fails with a 404 from OpenAI). Agent-S (agent="agents") does not implement it. With Agent-S, snapshotting still preserves the desktop and emits a token, but history_ref is always None and the agent starts the next run_once() with no memory of the prior conversation. By default this is logged at WARNING and the run continues (agent_trajectory_not_persisted on snapshot, agent_trajectory_not_supported on resume of a token that does carry history from another agent). Set strict_snapshot=True to turn both into a hard error instead — SnapshotError on snapshot, ResumeError on resume — rather than silently continuing with no agent memory; the sandbox is still torn down cleanly either way. A custom agent can opt in by setting supports_trajectory_persistence = True and wiring ctx._resume_history (seed) / ctx._captured_history (capture) itself.

Trajectory growth

Every resume re-feeds the full trajectory to the model, and it grows without bound by default. To keep cost and payload size in check:

• trajectory_max_images (GLCU_TRAJECTORY_MAX_IMAGES, default None = unbounded) — reserved for a future agent that implements image-capped trajectory replay. No built-in agent does today, so setting this to anything other than None raises ConfigError at config time rather than silently no-op'ing.
• trajectory_max_bytes (GLCU_TRAJECTORY_MAX_BYTES, default None) — a hard ceiling on the persisted trajectory JSON. When exceeded, the snapshot path fails before writing (raising SnapshotError under strict_snapshot, otherwise skipping history). Only takes effect for an agent that captures trajectory in the first place (see Agent support).
• trajectory_pii_anonymization (GLCU_TRAJECTORY_PII_ANONYMIZATION, default False) — reserved for a future agent that anonymizes PII within the captured trajectory itself. No built-in agent does today, so setting this to True raises ConfigError at config time. This is unrelated to GLCU_PII_REDACTION_ENABLED (see Observability), which is implemented and redacts PII from log lines, not from the trajectory replayed back to the model.

Provider semantics

• E2B takes true copy-on-write snapshots; predecessor snapshots are garbage-collected as a thread advances (set keep_snapshot_history=True to retain every snapshot for branching/forking).
• OpenSandbox pauses and resumes the same container — delete_snapshot is a no-op there. Paused threads accumulate and remain billable until explicitly released, so release sessions you no longer intend to resume.

Resuming a token whose model differs from the configured model raises ResumeError, because trajectory/image/caching formats differ across providers. Set allow_model_drift=true (GLCU_ALLOW_MODEL_DRIFT=true) to bypass that check at your own risk; a compatibility warning is logged.

---

Errors

All SDK exceptions extend GLComputerUseError:

• ConfigError — bad or missing credentials.
• SandboxProvisionError — the sandbox could not be allocated.
• GLTimeoutError — no event received within the configured timeout.
• TaskFailedError — the agent terminated with an error (TASK_FAILED).
• TaskCancelledError — the task was cancelled (TASK_CANCELLED).
• TakeoverRequiredError — on_takeover_needed returned resume=False (a deliberate cancellation) or raised (original exception chained as __cause__). Not raised when no callback is supplied; see Takeover.
• SnapshotError — a snapshot/history-persistence failure (only raised when strict_snapshot=True; see Snapshot & Resume).
• ResumeError — a token could not be resumed (e.g. model drift without allow_model_drift).

from gl_computer_use import (
    GLComputerUseClient,
    ConfigError,
    SandboxProvisionError,
    GLTimeoutError,
    TaskFailedError,
)

try:
    result = await GLComputerUseClient().run_once("do something", timeout=60.0)
except ConfigError as e:
    print("Check your API keys:", e)
except SandboxProvisionError as e:
    print("Sandbox failed to start:", e)
except GLTimeoutError as e:
    print("Took too long:", e)
except TaskFailedError as e:
    print("Agent failed:", e)

---

Observability

The SDK uses structlog for structured logging (JSON by default; set GLCU_LOG_FORMAT=console for human-readable output). Every line carries session_id, task_id, and component. Distributed tracing and metrics via OTLP, plus Sentry error tracking, are available through the observability extra and delegated to GDP Labs' gl-observability SDK. Optional regex-based PII redaction is enabled with GLCU_PII_REDACTION_ENABLED=true.

---

Custom Providers

Plug in alternative sandboxes, agents, or artifact stores without modifying the SDK:

from gl_computer_use import register_sandbox, GLComputerUseClient, GLComputerUseConfig
from gl_computer_use.sandbox.base import BaseSandbox


class MyCustomSandbox(BaseSandbox):
    ...  # implement abstract methods


register_sandbox("my-sandbox", MyCustomSandbox)
client = GLComputerUseClient(config=GLComputerUseConfig(sandbox="my-sandbox"))

register_agent and register_artifact work the same way for custom agents and artifact stores.

---

Local Development Setup

git clone git@github.com:GDP-ADMIN/gl-sdk.git
cd gl-sdk/libs/gl-computer-use
uv sync --all-extras
uv run gl-computer-use-setup
source .venv/bin/activate

Run checks:

uv run pytest           # tests
uv run ruff check .     # lint
uv run ruff check --fix # auto-fix lint
uv run mypy gl_computer_use/  # type-check

---

Contributing

Please refer to the Python Style Guide for code style, documentation standards, and SCA requirements.