⚡ See it work before you install — run the live browser benchmark →

Paste any URL. Watch Aliax shrink the page into a tiny, numbered map your model can actually read — side-by-side with the raw HTML, in seconds. No signup, no install.

---

Aliax — Python SDK

The production-grade reliability layer for VLM web agents.

Stop your AI agents from guessing browser coordinates and failing on responsive layouts. Aliax sits between your VLM and the live page, translates the DOM into a numbered Set-of-Mark image, executes the action the model picks deterministically, and ships every real-world failure into a labelled dataset you can fine-tune on.

It is the layer that turns "works in my notebook" into "holds up at 3am for paying corporate clients."

Python: 3.8 – 3.12 · Changelog · Contributing

Install

pip install aliax
playwright install chromium

aliax is the install name AND the import — no aliases, no version pinning required. No signup either: with no key, Aliax() bootstraps a free sandbox credential (500 parses; 100 from cloud egress) and caches it in ~/.aliax/credentials. Aliax() reads ALIAX_API_KEY from the environment so 12-factor deployments (Docker, Lambda, GitHub Actions) don't have to thread the key through code.

The 3-line agent loop

from aliax import Aliax, SYSTEM_INSTRUCTIONS
from playwright.async_api import async_playwright

async with Aliax() as aliax:                     # no key needed on first run
    async with async_playwright() as p:
        page = await (await p.chromium.launch()).new_page()
        await page.goto("https://shop.example.com/cart")

        # 1. Translate the live page into a Set-of-Mark image + node map.
        ctx = await aliax.parse_ui(page)

        # 2. Hand both to your VLM. Drop SYSTEM_INSTRUCTIONS into the system prompt.
        decision = await ask_llm(
            system=SYSTEM_INSTRUCTIONS,
            image=ctx.image_bytes,
            map=ctx.llm_text_block(),
        )
        # e.g. {"action": "CLICK", "element_id": "el_41"}

        # 3. Aliax executes it natively — scroll-into-view, React onChange,
        #    iframe coords, retina DPR, hydration races — all handled.
        await aliax.execute(page, decision)

That's the entire happy path. No XPath wrangling, no Playwright locator boilerplate, no pixel guessing.

---

Pillar 1 — The Steady Hand

Deterministic VLM-to-DOM translation. Your model never sees raw coordinates; it sees el_41 and Aliax handles the browser physics.

parse_ui(page) returns a ParseContext:

Attribute	Type	Description
image_bytes	bytes	Native Chromium screenshot (JPEG by default) with numbered Set-of-Mark boxes painted by the bundled JS overlay. Zero Python image processing — no Pillow, no OpenCV.
image_mime	str	"image/jpeg" by default, "image/png" if you opt in.
image_size	(int, int)	(width, height) in physical pixels, parsed from the image header.
elements	list[dict]	Each element: element_id, tag, role, text, bounds, editable, is_canvas, state, attrs, links_to. Only truly interactable nodes ≥12 CSS px.
viewport	dict	{width, height, dpr, scroll_x, scroll_y, page_scrollable_x, page_scrollable_y} in CSS px.
url	str	Page URL at capture time.
path	str	URL pathname only — e.g. "/settings/billing".
title	str	document.title at capture time.
truncated	bool	True when the DOM mapper hit its element cap; llm_text_block() appends a notice so the VLM knows the map is partial.

Convenience helpers: ctx.llm_text_block() (drop-in for your prompt) and ctx.route_context_block() (URL + title summary).

Execute — the full verb catalog

Every action execute() recognises:

await aliax.execute(page, {"action": "CLICK",          "element_id": "el_41"})
await aliax.execute(page, {"action": "HOVER",          "element_id": "el_14"})
await aliax.execute(page, {"action": "TYPE",           "element_id": "el_7", "value": "Nike Shoes"})
await aliax.execute(page, {"action": "TYPE_AND_ENTER", "element_id": "el_7", "value": "Nike Shoes"})
await aliax.execute(page, {"action": "PRESS",          "element_id": "el_7", "key": "Enter"})
await aliax.execute(page, {"action": "SCROLL_DOWN",    "dy": 600})       # also: dx / delta_x / delta_y
await aliax.execute(page, {"action": "SCROLL_UP",      "dy": 600})
await aliax.execute(page, {"action": "SCROLL_LEFT",    "dx": 400})
await aliax.execute(page, {"action": "SCROLL_RIGHT",   "dx": 400})
await aliax.execute(page, {"action": "NAVIGATE",       "url": "https://..."})
await aliax.execute(page, {"action": "WAIT",           "ms": 1500})
await aliax.execute(page, {"action": "COMBO",          "actions": [...]})  # compound action pipeline
await aliax.execute(page, {"action": "BATCH_TYPE",     "fields": [...]})   # multi-field fill in one turn
await aliax.execute(page, {"action": "REPORT_ISSUE",   "reason": "stuck"}) # routes through the Gatekeeper
await aliax.execute(page, {"action": "NOOP"})                              # terminal alias
await aliax.execute(page, {"action": "DONE"})                              # `FINISH` is accepted as an alias

Raw coordinates are accepted as an escape hatch:

await aliax.execute(page, {"action": "CLICK", "x": 905, "y": 150})

Under the hood, execute() scrolls the target into view, fires React-friendly events (focus → keystrokes with a 50ms cadence), refuses to click disabled controls, and never raises — your agent loop stays alive.

Token-aware rendering

VLM providers bill on file weight + dimensions. Drop to quality=40 for roughly 10× cheaper VLM calls without sacrificing Set-of-Mark ID legibility (the IDs are crisp DOM text rendered before the JPEG encoder runs).

ctx = await aliax.parse_ui(
    page,
    render_config={"format": "jpeg", "quality": 40},
)

Quality is clamped to [30, 100] so a typo like quality=5 can't turn the numbered boxes into illegible mush and crash your loop.

---

Pillar 2 — The Flight Recorder

Every agent eventually loops on a popup or hallucinates past recovery. Aliax ships two escalation entry points that double as your agentic telemetry stream:

report_issue() — the gated escalation

await aliax.report_issue(
    page,
    reason="submit_button_dead_zone",
    expected_outcome="navigate to /dashboard",
    actual_outcome="still on /login after 3 tries",
)

A strict Gatekeeper rejects the call unless the SDK's per-page failure history proves the agent is stuck:

• 3 identical state hashes of (url, spatial_map) → frozen DOM, and
• period-2 / 3 / 4 action cycle detection → radio-group / checkbox alternation loops the state hash is blind to.

Rejected calls never hit the API, never debit credits, and never flood the annotation queue with slow-network panic. Pass force=True only from a developer-side assertion (Playwright timeout, business invariant violation).

capture_failure() — the airbag

await aliax.capture_failure(
    page,
    goal="Close newsletter popup",
    thoughts=agent.current_reasoning,
    last_attempted_action={"action": "CLICK", "element_id": "el_41"},
    failure_reason="modal_blocked",
    step=agent.loop_step,
)

POST /v1/capture with idempotency UUID, 3 attempts and exponential backoff on 429/502/503/504, automatic transport-error fallback. The full state — screenshot, DOM map, viewport, the agent's own reasoning trail — lands in the Aliax Inbox for human triage.

---

Pillar 3 — The Closed Loop

Every captured failure becomes a (negative, positive) DPO pair: the agent's wrong move plus the annotator's corrected tap. Export the dataset and fine-tune; your next deployment fails less often on the exact failure modes that bit you in production.

That's the loop:

parse_ui → VLM → execute → (95% happy path)
                            ↓ stuck
                       report_issue → Inbox → DPO dataset → fine-tune

---

Architecture

1. The bundled encrypted DOM mapper (dom-mapper.dat, shipped inside the wheel) is opened only after the authenticated licensing session, then walks the live page including shadow DOMs and same-origin iframes, returning only interactable elements ≥12 CSS px.
2. The same JS module paints a position:fixed; pointer-events:none; contain:strict overlay of numbered boxes. The host page's layout / hover / IntersectionObserver state is untouched.
3. Playwright snaps one native screenshot via its C++ CDP path at the format / quality you asked for. No Python pixel processing.
4. The overlay is torn down in a try/finally so a mid-capture crash leaves the live page exactly as we found it.
5. Coordinates are cached so execute(page, {"element_id": "el_41"}) resolves instantly against the most recent parse_ui().
6. Every call fires a non-blocking telemetry ping for billing.

Hardening for long-running agents

• async with Aliax() as aliax: — full async context manager support so the httpx connection pool is torn down deterministically even if the loop crashes. No socket leaks in 24/7 bots.
• Zip-safe asset loading — dom-mapper.dat is loaded via importlib.resources, so the SDK runs inside AWS Lambda layers and Cloud Run images where __file__ is virtual.
• PEP 561 typed — ships py.typed; mypy, Pyright, VS Code and PyCharm consume the inline type hints out of the box.
• Automatic *.workers.dev fallback — if the apex DNS / TLS flakes, the SDK transparently retries against the worker origin. Opt out with fallback_endpoint="".
• Self-healing billing latches — HTTP 402 sets credits_exhausted and clears on any successful 200. HTTP 401 latches invalid_key_reason (terminal — mint a new key).

Public surface

from aliax import (
    Aliax,
    ParseContext,
    AttemptedAction,
    AliaxError,
    AliaxOutOfCreditsError,
    AliaxInvalidKeyError,
    ALIAX_SYSTEM_INSTRUCTIONS,  # primary — drop into your VLM system prompt
    SYSTEM_INSTRUCTIONS,        # short alias for the same string
)

Aliax constructor (all keyword-only, all optional):

Kwarg	Default	Notes
api_key	os.getenv("ALIAX_API_KEY"), else a free anonymous sandbox key	Must start with sk_.
allow_anonymous	True	With no key, bootstrap a free machine-scoped sandbox credential (cached in ~/.aliax/credentials): 500 parses on a home/office network, 100 from cloud egress, 30-day life, no crash captures. Set False (or ALIAX_DISABLE_ANONYMOUS=1) to hard-fail instead.
endpoint	https://api.aliax.xyz/v1	Override for self-hosted.
fallback_endpoint	workers.dev origin (auto)	Pass "" to disable.
debug_mode	False	Writes payloads locally instead of POSTing.
redact_selectors	[]	Merged with built-in PII baseline (passwords, emails, card numbers).
check_for_updates	True	Pings /v1/version once at init; pass False in airgapped envs.
max_image_dim	0	Deprecated — no-op since v1.0. Use render_config={"quality": N} on parse_ui for VLM token economy. Kept in the signature so pre-1.0 callers keep importing; a one-shot deprecation log fires only when explicitly set.

Methods: parse_ui, execute, report_issue, capture_failure, get_element_coords, billing_status, refresh_billing_status, aclose.

---

Get an API key · Docs · Inbox