aliax 1.0.2

Aliax SDK — real-time interceptor middleware for visual web agents (Set-of-Mark + deterministic execution).

Aliax SDK

Aliax is the real-time interceptor middleware for visual web agents. Translate the live page into a numbered Set-of-Mark image, hand it to your VLM, and let Aliax execute the action it picks — clicks, types, scrolls, the works — with deterministic accuracy.

When the agent still gets stuck, ship the failure to the offline annotation queue with capture_failure().

Current version: 1.0.2 · Python: 3.8 – 3.12 · Changelog · Contributing

Installation

pip install aliax
playwright install chromium

The 3-line agent loop

from aliax import Aliax
from playwright.async_api import async_playwright

aliax = Aliax(api_key="sk_live_...")   # mint one at /api-keys

async with async_playwright() as p:
    page = await (await p.chromium.launch()).new_page()
    await page.goto("https://shop.example.com/cart")

    # 1. Translate the live DOM into a VLM-friendly Set-of-Mark.
    ctx = await aliax.parse_ui(page)

    # 2. Ask your VLM. It sees a numbered screenshot + a tiny JSON map.
    decision = await ask_llm(image=ctx.image_bytes, map=ctx.elements)
    # e.g. {"action": "CLICK", "element_id": "el_41"}

    # 3. Aliax executes it natively — scroll-into-view, React onChange,
    #    iframe coords, retina DPR, all handled.
    await aliax.execute(page, decision)

That's the entire happy path. No XPath wrangling, no Playwright boilerplate, no pixel guessing.

The safety net — capture_failure()

When the agent loops on a popup or hallucinates past recovery, snapshot the full state and ship it to the Aliax annotation queue for human correction:

await aliax.capture_failure(
    page,
    goal="Close newsletter popup",
    thoughts=agent.current_reasoning,
    last_attempted_action={"action": "CLICK", "element_id": "el_41"},
    failure_reason="modal_blocked",
    step=agent.loop_step,
)

Each failure becomes a (negative, positive) DPO pair — the agent's "stupid move" + the annotator's corrected tap — ready for fine-tuning.

parse_ui() — what you get back

ctx = await aliax.parse_ui(page)

Attribute	Type	Description
image_bytes	bytes	Native browser screenshot (JPEG by default — see render_config) with numbered Set-of-Mark boxes painted by the bundled JS overlay. Zero Python image processing.
image_mime	str	"image/jpeg" by default, "image/png" if you opt in.
image_size	(int, int)	(width, height) in physical pixels, parsed from the image header.
elements	list[dict]	Each element: element_id, tag, role, text, bounds, editable, is_canvas, attrs.
viewport	dict	{width, height, dpr, scroll_x, scroll_y} in CSS px.
url	str	Page URL at capture time.

Power-user: shrink your VLM bill with render_config

VLM providers bill image tokens by file weight + dimensions. JPEG at quality=40 costs roughly 10× fewer tokens than the lossless default. The Set-of-Mark IDs stay readable because they're rendered as crisp DOM text by Chromium before the JPEG encoder runs.

# Default — pristine JPEG at q=80.
ctx = await aliax.parse_ui(page)

# Token-optimised — same readable IDs, ~10× cheaper per VLM call.
ctx = await aliax.parse_ui(
    page,
    render_config={"format": "jpeg", "quality": 40},
)

# Lossless when you need it (research / fine-tuning datasets).
ctx = await aliax.parse_ui(
    page,
    render_config={"format": "png"},
)

Quality is clamped to [30, 100] so a typo like quality=5 can't turn the numbered boxes into illegible mush and crash your agent loop.

Convenience helper:

prompt = f"""
{ctx.llm_text_block()}
Pick the element_id to click. Reply JSON {{action, element_id}}.
"""

execute() — the action verbs

await aliax.execute(page, {"action": "CLICK",    "element_id": "el_41"})
await aliax.execute(page, {"action": "TYPE",     "element_id": "el_7", "value": "Nike Shoes"})
await aliax.execute(page, {"action": "HOVER",    "element_id": "el_14"})
await aliax.execute(page, {"action": "PRESS",    "element_id": "el_7", "key": "Enter"})
await aliax.execute(page, {"action": "SCROLL",   "dy": 600})
await aliax.execute(page, {"action": "NAVIGATE", "url": "https://..."})
await aliax.execute(page, {"action": "WAIT",     "ms": 1500})
await aliax.execute(page, {"action": "DONE"})

Raw coordinates are also accepted as an escape hatch:

await aliax.execute(page, {"action": "CLICK", "x": 905, "y": 150})

Under the hood, execute() scrolls the target into view, fires React-friendly events (focus → keystrokes with a 50ms cadence), and returns a small dict like {"ok": True, "coords": [905, 150]}. It never raises — your agent loop stays alive.

How it works under the hood

1. The bundled DOM mapper (dom-mapper.min.js, shipped inside the wheel) traverses the live page including shadow DOMs and same-origin iframes. It returns only truly interactable elements with a minimum size of 12 CSS pixels — no clutter, no hallucinated boxes.
2. The same JS module paints a position:fixed; pointer-events:none; contain:strict overlay of numbered colored boxes at the top of the z-stack. Chromium renders the boxes in microseconds and the host page's layout / hover / IntersectionObserver state is untouched.
3. Playwright snaps one native screenshot via its C++ CDP path at the format / quality you asked for in render_config. No Python image processing, no Pillow dependency.
4. The overlay layer is torn down in a try/finally so a crash mid- capture still leaves the live page exactly as we found it.
5. Coordinates are cached so execute(page, {"element_id": "el_41"}) resolves instantly against the most recent parse_ui().
6. Every call fires a non-blocking telemetry ping for usage tracking.

What's new in v1.0.2 — version parity with Node SDK

The 1.0.0 cut is the first official PyPI release. The SDK has been hardened for enterprise deployment — zipped wheels, serverless layers, long-running agents — and the install name now matches the import (pip install aliax → import aliax).

• async with Aliax() as aliax: — full async context manager support so the httpx connection pool is torn down deterministically even if the agent loop crashes. No more socket leaks in 24/7 bots.
• ALIAX_API_KEY env-var fallback — Aliax() with no args reads the key from the environment; 12-factor Docker / Lambda / GitHub Actions deployments don't have to thread it through code.
• Zip-safe asset loading — dom-mapper.min.js is read via importlib.resources, so the SDK runs inside AWS Lambda layers and Google Cloud Run container images where __file__ is a virtual path.
• PEP 561 typed — ships py.typed so mypy, Pyright, VS Code, and PyCharm consume the inline type hints out of the box.
• PEP 621 packaging — pure pyproject.toml, dynamic version pulled from aliax/_version.py, MIT-licensed, classified as Development Status :: 5 - Production/Stable.
• render_config={"format", "quality"} — the only knob AI engineers get for VLM token economics. Defaults are pristine JPEG at quality=80; drop to quality=40 for roughly 10× cheaper VLM calls without sacrificing Set-of-Mark ID legibility. Quality is clamped to [30, 100] so a typo can't crash the agent.
• Browser-native overlay — Set-of-Mark boxes are painted by the bundled dom-mapper.min.js as a position:fixed; pointer-events:none; contain:strict DOM layer. Chromium renders them in microseconds; Playwright snaps the screenshot via its native C++ CDP path. The SDK never touches the pixel buffer — no Pillow, no OpenCV, no Cairo.
• parse_ui(page) / execute(page, decision) / capture_failure(...) — the canonical 3-call agent loop, with a typed ParseContext, AttemptedAction dataclass, and the full action catalog (CLICK, TYPE, TYPE_AND_ENTER, HOVER, SCROLL_{DOWN,UP,LEFT,RIGHT}).
• Hardened transport — bearer-only auth, idempotency keys on capture_failure, race-safe HTTP client init, non-blocking usage telemetry, automatic *.workers.dev fallback if the apex DNS / TLS flakes, opt-out via fallback_endpoint="".

See CHANGELOG.md for the full changelog.