uisurf-agent

uisurf-agent is a Python package for running UI automation agents against:

• a web browser via Playwright
• the local desktop via desktop control utilities
• an Android device or emulator via ADB and adbutils

The project includes:

• a package-level Typer CLI at src/uisurf_agent/cli.py
• A2A server entrypoints for browser, desktop, and mobile agents
• interactive local execution for manual testing

Requirements

• Python 3.11+
• pip
• Node.js only if you also use the bundled a2a-inspector
• Playwright browser binaries installed for browser automation
• a desktop environment if you run the desktop agent
• Android platform tools plus an authorized ADB device or emulator if you run the mobile agent

Install

Install the package from PyPI:

pip install uisurf-agent

Install Playwright browsers if needed:

python -m playwright install

For local development from a source checkout, use uv instead:

uv sync
uv run playwright install

Environment

The package reads configuration from environment variables and .env files via python-dotenv.

Common variables:

• UISURF_PROVIDER: registered provider name. Default is gemini.
• UISURF_MODEL_ID / MODEL_ID: provider-specific model identifier. Gemini defaults to gemini-3-flash-preview.
• UISURF_PROVIDER_BASE_URL / UISURF_MODEL_BASE_URL / MODEL_BASE_URL: optional provider-specific model base URL.
• AGENT_HOST: default bind host for A2A servers
• BROWSER_AGENT_PORT: default browser A2A port, default 8001
• DESKTOP_AGENT_PORT: default desktop A2A port, default 8002
• MOBILE_AGENT_PORT: default mobile A2A port, default 8003
• BROWSER_AGENT_PUBLIC_URL: public URL advertised by the browser A2A server
• DESKTOP_AGENT_PUBLIC_URL: public URL advertised by the desktop A2A server
• MOBILE_AGENT_PUBLIC_URL: public URL advertised by the mobile A2A server
• BROWSER_FAST_MODE: speeds up browser settling by waiting less aggressively
• AUTO_MODE: automatically approve safety-gated A2A actions by default
• BROWSER_AGENT_AUTO_MODE: browser A2A override for automatic safety approval
• DESKTOP_AGENT_AUTO_MODE: desktop A2A override for automatic safety approval
• MOBILE_AGENT_AUTO_MODE: mobile A2A override for automatic safety approval
• INCLUDE_THOUGHTS: global default for model thought streaming when supported
• BROWSER_INCLUDE_THOUGHTS: browser-only override for thought streaming
• DESKTOP_INCLUDE_THOUGHTS: desktop-only override for thought streaming
• MOBILE_INCLUDE_THOUGHTS: mobile-only override for thought streaming
• BROWSER_AGENT_MAX_STEPS: browser A2A step limit, default 40
• DESKTOP_AGENT_MAX_STEPS: desktop A2A step limit, default 40
• MOBILE_AGENT_MAX_STEPS: mobile A2A step limit, default 40
• DESKTOP_OBSERVATION_DELAY_MS: delay before each desktop screenshot capture
• MOBILE_OBSERVATION_DELAY_MS: delay before each mobile screenshot capture
• MOBILE_DEVICE_SERIAL: Android device serial used by mobile A2A when --mobile-serial is omitted
• MAX_OBSERVATION_IMAGES: number of screenshot observations that keep image payloads in history
• OBSERVATION_SCALE: default screenshot scale, from 0 < scale <= 1
• BROWSER_OBSERVATION_SCALE: browser-only screenshot scale override
• DESKTOP_OBSERVATION_SCALE: desktop-only screenshot scale override
• MOBILE_OBSERVATION_SCALE: mobile-only screenshot scale override

Screenshot scaling only changes the image sent to the model. Action coordinates still map to the full browser viewport, desktop resolution, or Android device resolution.

When using the built-in gemini provider, you will also need the credentials required by the Google client.

CLI

The CLI is implemented with Typer and is exposed through both:

uisurf-agent --help

Current top-level command:

uisurf_agent run --help

Convenience make targets are also available for local A2A server runs:

make run-browser
make run-desktop
make run-mobile
make run-dev
make run-all

These targets start long-running local servers instead of one-off tasks. By default they bind to:

browser: http://127.0.0.1:8001/
desktop: http://127.0.0.1:8002/
mobile: http://127.0.0.1:8003/

You can override the bind host and ports from the shell:

make run-browser HOST=127.0.0.1 BROWSER_PORT=8001
make run-desktop HOST=127.0.0.1 DESKTOP_PORT=8002
make run-mobile HOST=127.0.0.1 MOBILE_PORT=8003 MOBILE_SERIAL=5C060DLCR002MM MOBILE_OBSERVATION_DELAY_MS=1500
make run-dev HOST=127.0.0.1 BROWSER_PORT=8001 DESKTOP_PORT=8002 MOBILE_PORT=8003
make run-all HOST=127.0.0.1 BROWSER_PORT=8001 DESKTOP_PORT=8002 MOBILE_PORT=8003

If one of those ports is already in use, the make target will exit early with a clear message. Rerun with free ports if needed.

If you want the previous one-off local task mode, use the interactive targets:

make run-browser-interactive TASK="Open example.com and summarize the page"
make run-desktop-interactive TASK="Open Terminal and run pwd"

Run the browser agent interactively

uisurf_agent run browser_agent \
  --task "Open example.com and summarize the page" \
  --fast-mode \
  --no-include-thoughts \
  --max-observation-images 2 \
  --observation-scale 0.75 \
  --max-steps 20

Run headless:

uisurf_agent run browser_agent \
  --headless \
  --task "Go to Hacker News and summarize the top 5 stories"

Select a registered provider:

uisurf_agent run browser_agent \
  --provider gemini \
  --model-id gemini-3-flash-preview \
  --task "Open example.com and summarize the page"

Extending providers

Provider implementations and custom model registration are documented in mkdocs/models/external-models.md.

Extending browser, desktop, and mobile tools

Browser, desktop, and mobile custom tools use the same registry pattern. A declaration function describes the tool schema sent to the model. An optional handler executes the tool when the selected agent receives that function call:

from typing import Any

from uisurf_agent import BrowserAgent, register_browser_tool


async def save_page_title(agent: BrowserAgent, args: dict[str, Any]) -> None:
    title = await agent._browser_controller.page.title()
    print(f"{args['label']}: {title}")


@register_browser_tool(handler=save_page_title)
def save_page_title(label: str) -> dict[str, str]:
    """Save the current page title with a label."""
    return {"label": label}

Use register_desktop_tool(...) for desktop-only tools, register_mobile_tool(...) for Android-only tools, or register_tool("browser" | "desktop" | "mobile", ...) when the environment is selected dynamically. If a registered tool has no handler, the agent falls back to a controller method with the same name. Installed packages can expose tool setup through the uisurf_agent.tools entry-point group. Entry points may expose a registration function, a ToolRegistration, or a declaration callable named like browser.my_tool, desktop.my_tool, or mobile.my_tool.

Run the desktop agent interactively

uisurf_agent run desktop_agent \
  --task "Open Terminal and run pwd" \
  --desktop-observation-delay-ms 750 \
  --no-include-thoughts \
  --max-observation-images 2 \
  --observation-scale 0.75 \
  --max-steps 10

Automatically approve safety-gated actions:

uisurf_agent run desktop_agent \
  --task "Open a text editor and type Hello World" \
  --auto-mode

Run the mobile agent interactively

First verify that ADB can see your Android device or emulator:

adb devices -l

Then run MobileAgent:

uisurf_agent run mobile_agent \
  --task "Open Android Settings and tell me what is visible" \
  --mobile-serial 5C060DLCR002MM \
  --auto-mode \
  --max-steps 10

If --mobile-serial is omitted, the controller uses the first authorized ADB device.

For Python API usage, see mkdocs/agents/mobile.md.

A2A Server Mode

Browser, desktop, and mobile agents can be exposed as A2A servers through the CLI.

Browser A2A server

uisurf_agent run browser_agent \
  --mode a2a \
  --host 0.0.0.0 \
  --port 8080

If --port is omitted, the browser agent defaults to 8001.

Desktop A2A server

uisurf_agent run desktop_agent \
  --mode a2a \
  --host 0.0.0.0 \
  --port 8081

If --port is omitted, the desktop agent defaults to 8002.

Mobile A2A server

uisurf_agent run mobile_agent \
  --mode a2a \
  --host 0.0.0.0 \
  --port 8003 \
  --mobile-serial 5C060DLCR002MM

If --port is omitted, the mobile agent defaults to 8003. If --mobile-serial is omitted, the server uses MOBILE_DEVICE_SERIAL or the first authorized ADB device.

MCP Mode

The CLI accepts --mode mcp, but MCP server mode is not implemented yet. The command currently exits with a clear error instead of starting a server.

Docker

The repository includes a containerized runtime that starts:

• noVNC on port 6080
• a Chromium instance inside the container with remote debugging on port 9222
• the browser agent A2A server on port 8001
• the desktop agent A2A server on port 8002

Provide environment variables

The recommended approach is to place secrets and runtime settings in a local .env file in the repository root. The wrapper script run.sh will automatically pass that file to Docker with --env-file if it exists.

Example .env:

GEMINI_API_KEY=your_key_here
UISURF_PROVIDER=gemini
UISURF_MODEL_ID=gemini-3-flash-preview
AGENT_HOST=0.0.0.0
BROWSER_AGENT_PORT=8001
DESKTOP_AGENT_PORT=8002
BROWSER_FAST_MODE=true
AUTO_MODE=false
INCLUDE_THOUGHTS=false
DESKTOP_OBSERVATION_DELAY_MS=750
MAX_OBSERVATION_IMAGES=2
OBSERVATION_SCALE=0.75
PASSWORD_REQUIRED=false

You can also point the wrapper at a different file:

ENV_FILE=.env.local sh ./run.sh

Build and run the container

From the repository root:

sh ./run.sh

The script builds the image from docker/Dockerfile, starts the container, and publishes the default ports to the host.

Default host endpoints:

• noVNC: http://localhost:6080
• browser A2A server: http://localhost:6080/browser/
• desktop A2A server: http://localhost:6080/desktop/

Override published ports

The wrapper script supports environment variable overrides:

BROWSER_AGENT_PUBLIC_URL=http://localhost:6081/browser/ \
DESKTOP_AGENT_PUBLIC_URL=http://localhost:6081/desktop/ \
PORT=6081 \
sh ./run.sh

View logs

docker logs -f uisurf-agent-test

Stop the container

docker rm -f uisurf-agent-test

Notes

• The container startup will fail early if neither GEMINI_API_KEY nor GOOGLE_API_KEY is provided.
• VNC password auth is disabled by default. Set PASSWORD_REQUIRED=true if you want the frontend to require a VNC password again.
• Inside the container, Chromium is started separately and the browser controller connects to it over CDP at http://127.0.0.1:9222.
• The desktop and browser agents are both started through the package CLI from src/uisurf_agent/cli.py.
• MobileAgent is available from the package CLI/Python API and A2A mode, but the default Docker runtime does not start a mobile A2A server or provision an Android emulator/device.

Logging

The package configures a Rich-backed library logger in src/uisurf_agent/init.py. The logger name is uisurf_agent.

Package Layout

• src/uisurf_agent/cli.py: Typer CLI
• src/uisurf_agent/a2a/browser_a2a.py: browser A2A server
• src/uisurf_agent/a2a/desktop_a2a.py: desktop A2A server
• src/uisurf_agent/a2a/mobile_a2a.py: mobile A2A server
• src/uisurf_agent/agents/browser_agent.py: browser automation agent
• src/uisurf_agent/agents/desktop_agent.py: desktop automation agent
• src/uisurf_agent/agents/mobile_agent.py: Android automation agent
• src/uisurf_agent/utils/mobile_controller.py: ADB-backed mobile controller

Development

Validate the package modules compile:

uv run python -m compileall src/uisurf_agent

Show CLI help locally through the package module:

uv run python -m uisurf_agent --help