Qirabot

English | 简体中文

Cross-platform GUI automation, driven by multimodal AI vision. Drive browsers, mobile apps, full desktops, and games through pixels — no DOM, no selectors — reaching what frameworks like Playwright, Selenium, and Appium cannot.

Run it standalone (bot.open() launches a browser for you; Android / iOS / Windows-window backends are built in with zero extra dependencies), bolt it onto your existing Playwright / Selenium / Appium / pyautogui session, drop it into a pytest suite, or bind by HWND to drive a Unity / Unreal / native desktop game. Same API across all of them.

The decision engine runs in your own process: each screenshot goes directly from your machine to a vision model you configure on Google Vertex AI, authenticated with your own Google Cloud credentials and billed by Google.

📖 Full documentation: qirabot.com/docs (中文)

See it work

https://github.com/user-attachments/assets/649ea80c-63e7-4c85-9ee8-3c8fe17e5ef4

Play an MMORPG from zero to level 15, hands-free — iOS real device. The entire task prompt is one sentence: "This is Fantasy Westward Journey mobile. Create a character, then complete the new-player flow; skip whatever can be skipped." Highlights cut from a single unedited run: full 5:50 video · script

More real, unedited runs — the AI sees only pixels. Click a poster to watch (all demos →):

Clear AFK Journey's tutorial and reach the open world — iOS real device

Play chess on lichess.org — Android real device

Beat a fruit tile-match game on its own — Android real device

Installation

One line — installs uv, qirabot (isolated, never touches your system Python), and Chromium. No pre-installed Python required:

# macOS / Linux
curl -LsSf https://qirabot.com/install | sh

# Windows (PowerShell)
powershell -ExecutionPolicy ByPass -c "irm https://qirabot.com/install.ps1 | iex"

Driving a device instead of a browser? The Android (adb), iOS (WDA), and Windows single-window backends are built into the core package:

uv tool install qirabot        # Android + iOS + Windows window; zero extras

pip, virtualenvs, per-framework extras, and troubleshooting: Installation guide. Whichever path you took, qirabot doctor reports what is installed, what is missing (with the exact fix), and whether your Google Cloud credentials resolve.

Quick Start

Qirabot calls a vision model on Google Vertex AI with your own Google Cloud credentials. Set up Application Default Credentials (ADC) once:

gcloud auth application-default login
# or, for service accounts / CI:
# export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json

No gcloud setup? For gemini-vertex models a Vertex AI API key works instead of ADC — note this is a Google Cloud API key, not an AI Studio key. Google models only, global endpoint only:

export QIRA_VERTEX_API_KEY="..."   # or Qirabot(vertex_api_key=...)

No Google Cloud at all? The gemini provider calls the Gemini Developer API with an AI Studio API key:

export QIRA_GEMINI_API_KEY="..."   # or GEMINI_API_KEY, or Qirabot(gemini_api_key=...)
export QIRA_MODEL="gemini/gemini-3.8-flash"

Optionally pick a model as {provider}/{model} — provider is one of gemini-vertex, gemini; the default is gemini-vertex/gemini-3.8-flash. Set the Google Cloud project if your credentials don't already carry one:

export QIRA_MODEL="gemini-vertex/gemini-3.8-flash"   # or Qirabot(model=...)
export QIRA_VERTEX_PROJECT="my-gcp-project"                # or Qirabot(vertex_project=...)

qirabot models lists the providers and verifies your credentials. Then hand the AI a task. Real, unedited output:

$ qirabot browser "Search for SpaceX and get the first sentence of the article" --url wikipedia.org
Run: 6237d4ff
[1/20] type_text  ← "SpaceX"
        └ Type 'SpaceX' into the Wikipedia search bar and press enter to search.
Done: Space Exploration Technologies Corp., doing business as SpaceX, is an
      American spaceflight, telecommunications, and artificial intelligence
      company headquartered at the Starbase development site in Starbase, Texas.

Every run writes an HTML report with per-step screenshots; --record captures a video of the whole run.

For sites that need an account, log in once by hand — no site credentials in your scripts — and every later run that reuses the profile starts already signed in:

qirabot open-browser --user-data-dir ~/.automation --url news.ycombinator.com/login
# log in in the window, close it, then:
qirabot browser "Upvote the top story about Rust" --user-data-dir ~/.automation

Python SDK

The CLI is powered by the same engine. Call bot.ai() from Python and the AI likewise looks at the screen, decides the next action, and loops until the task is done — except the result lands directly in your code, with an on_step callback streaming each action as it happens:

from qirabot import Qirabot, StepResult

bot = Qirabot()
page = bot.open("https://www.wikipedia.org")

def on_step(step: StepResult) -> None:
    label = "done" if step.finished else step.action_type
    print(f"  step {step.step}: {label} {step.params}")

result = bot.ai(page, "Search for SpaceX and get the first sentence of the article", on_step=on_step)
print(f"Success: {result.success}")
print(f"Result: {result.output}")

bot.close()

Prefer to drive each step yourself? The same natural-language targeting works as single-step calls — bot.click(page, "Login button"), bot.extract(...), bot.verify(...) — with your code in control. Need just the coordinates? x, y = bot.locate(page, "the OK button") resolves an element without acting, so you can feed them to your own framework calls.

Bolt onto your existing stack

No rewrite: pass your existing page / driver / device object and mix AI steps with the selectors you already have. Add AI where selectors hurt — visual assertions, dynamic widgets, and flows too tedious to script:

import pytest
from qirabot import Qirabot

@pytest.fixture(scope="session")
def bot():
    with Qirabot(task_name="test-checkout") as bot:   # one task per run
        yield bot

def test_checkout(page, bot):     # `page` is your pytest-playwright fixture
    page.goto("https://shop.example.com")
    page.fill("#username", "test_user")             # your selectors, as-is
    page.click("#login-btn")

    # Visual assertion — survives markup rewrites and CSS refactors
    assert bot.verify(page, "the product grid shows items with prices and no error banner")

    # One line replaces a page of brittle selector steps
    result = bot.ai(page, "Complete checkout, name John Doe zip 10001", max_steps=8)
    assert result.success

Works the same for Selenium, Appium, pyautogui, and the built-in device backends (AdbDevice, WdaClient, Window) — and anything else via a 7-primitive custom adapter.

Use with AI agents

Qirabot ships a pre-built skill following the Agent Skills open standard, loadable by Claude Code, Codex, Cursor, and other compatible agents. The skill bundles a preflight environment check, a condensed API reference aligned with the SDK version, and per-platform starter templates. Given a natural-language automation goal, the agent validates the environment, picks the execution path — the CLI for one-shot tasks, an SDK script when the flow needs branching or returned values — and verifies the run's outcome.

Any Agent-Skills-compatible tool can install the copy bundled in the pip package, version-matched to the installed SDK:

qirabot skill install agents        # or claude / codex / cursor / --dir <path>

Claude Code users should prefer the plugin marketplace (auto-updates):

/plugin marketplace add qirabot/claude-plugins
/plugin install qirabot@qirabot

The skills CLI also works, installing from the repository's main branch:

npx skills add qirabot/qirabot

The skill's reference and templates are drift-tested against the live SDK in CI (tests/test_skill.py). Details: plugins/qirabot/README.md.

Domain knowledge: teach the AI your rules

The model knows how to drive a UI — not your game's item names or your team's business terms. Mount reference text for the task and the AI consults it at every step. From the CLI, -k takes a file and repeats, 32KB total:

qirabot browser "Buy 10 stamina potions in the shop" -k game-rules.md -k gm-policy.md

From Python, knowledge takes literal text, a UTF-8 file, or a list mixing both:

result = bot.ai(
    device,
    "Complete every daily quest",
    knowledge=[Path("game-rules.md"), "GM commands may be used once per match"],
)

Knowledge is mounted per call: the next bot.ai() starts clean, so each stage of a long flow carries only what it needs. Two deliberate limits: no URLs — fetch remote sources yourself, so auth and failures stay in your code — and knowledge guides decisions; hard rules like "once per match" belong in custom-tool code (next section), where they can actually be enforced.

Custom tools: let the AI call your code

Mid-task, the AI isn't limited to clicking and typing. custom_tools registers plain Python functions the model can invoke as it works — hit an internal API, query a database, fetch an OTP from your mail server, seed test data, or pause for a human at a CAPTCHA. Name, description, and parameters are introspected from the function itself:

def gm_command(command: str) -> str:
    """Send a command to the game's GM backend and return its reply.
    Available commands: add_energy <amount>, add_gold <amount>"""
    return requests.post(GM_URL, json={"cmd": command}, timeout=10).text

result = bot.ai(
    device,
    "Complete every daily quest. If an out-of-energy popup appears, "
    "use gm_command to add 100 energy and continue",
    custom_tools=[gm_command],
)

The tool runs in your own process, like everything else qirabot does — the model sees only the tool's name, description, parameters, and return value, never your endpoints or credentials — and that return value becomes the model's next observation. One instruction now spans systems that used to take a page of glue code: UI steps, backend calls, and human handoffs in a single flow. Details (schemas, error handling, pruning built-in tools): AI Tasks & Custom Tools. Runnable examples: custom_tool_gm.py · 06_human_in_the_loop.py.

The two parameters together also cover instructions you already maintain for another agent — an agent skill (SKILL.md), a runbook: mount the file with knowledge=, and for CLI steps it contains, register a whitelisted run_command custom tool (built-in tools are deliberately GUI-only). Pattern and reference implementation: Reusing an existing skill.

Progress overlay

Every CLI task command shows a small always-on-top window in the screen's bottom-right corner: the running instruction, each step's action and reasoning, and the final ✓/✗ outcome. The window is excluded from screen capture and click-through — it never appears in the bot's own screenshots and never intercepts a click meant for the app below. Turn it off with --no-overlay; in the SDK it's one flag:

bot = Qirabot(overlay=True)   # every bot.ai() run reports to the window

When a task drives the machine's real mouse and keyboard (the desktop backends), a slow-breathing amber glow lines the screen edges — the "machine is being controlled, hands off" signal — and holding ESC for about a second aborts the run: the bot releases every key it was holding and the task is recorded as cancelled, not failed. Remote-protocol targets (browser, Android, iOS) light neither: your mouse stays yours there.

A standalone Overlay also displays your own script's phases alongside the bot's steps. Details — abort semantics (user_abort, clear_user_abort()), the standalone API, platform notes and limits: Progress Overlay & Kill Switch. Runnable example: overlay_progress.py.

Documentation

Examples

Runnable examples live in examples/, in three styles:

• Bolt onto your existing tests (pytest) — playwright/, selenium/, appium/, desktop/
• Standalone automation (plain scripts) — scraping / RPA / agents: automation/
• Drive a game — Windows desktop games (bind by HWND) and the iOS MMORPG script behind the demo video: game/

See examples/README.md for which to pick.

License

MIT