hugpy-browser

Screen-driven browser control for bots — no Selenium, no WebDriver, no debugging port — standalone and trainable. The tools look at the screen and move the real pointer, so a site sees an ordinary user's browser. Measured: pointer error 0 px on VM and desktop; description aims 11/12 on target, 1 px median error (Qwen2.5-VL-7B).

This package is self-contained. The engine is inside it — hugpy_browser.engine is a verbatim copy of the abstract-toolserver 0.0.70 browser stack (browser.py, aim.py, surfaces.py, labcheck.py, vl.py, vmgui.py, uitools.py, plus a minimal tools.py shim). Nothing was removed from abstract-toolserver; the toolserver still ships and serves the same code. Provenance is recorded in engine/__init__.py.

Install

pip install hugpy-browser            # engine + docs + data (pillow only)
pip install 'hugpy-browser[full]'    # + OCR, OpenCV snapping, desktop input, renderer, click macros

Extras map to features (everything degrades gracefully): [ocr] text= aims and browser_look; [cv] region OCR pass + element snapping + lab checks; [desktop] on="screen" capture/input; [render] headless page render fallback; [clicks] the lab's qemu/firefox macros. Description (target=) aims need a reachable vision model (HUGPY_BASE, or any OpenAI-style endpoint for the bench).

Quickstart

Through a toolserver the tools are served as browser_*; as a library:

from hugpy_browser import browser

browser.browser_go(url="https://example.com", on="vm:ubuntu-desktop")
items = browser.browser_look()["items"]
browser.browser_click(text="Sign in")
browser.browser_type(text="me@example.com", into_text="Email", submit=True)

Aim four ways: text= (OCR, exact visible words), target= (vision-model description, confirmed on a magnified crop), cell="J2.D5" (labelled grid for a bot that reads images itself), or x=, y=. Every click is verified: before/after frames, diff regions, a verdict (changed_at_click | changed_elsewhere | no_change).

Training

The engine aims descriptions with the VL model's native grounding — the prompts in engine.aim._point_prompt, answers {"bbox_2d": [x1, y1, x2, y2]}. The exporter emits exactly that distribution from the bundled target set:

hugpy-browser dataset --out /tmp/ds --format qwenvl --zoom-aug --wire
# -> train.jsonl (chat-messages records: image + grounding prompt -> bbox JSON)
#    screen views, the engine's magnified confirm views, and wire-geometry views

Generate more synthetic pages with bench/make_bench.py (playwright), then re-score the tuned model with bench/evalrun.py against any OpenAI-style endpoint (bench/vlclient.py, env VL_BASE). Results of record: docs/final_numbers.md.

The lab

A repeatable desktop for tests: libvirt image hugpy-browser-lab (Firefox opens on the bundled precision page). browser_lab(action="start") boots an ephemeral clone; browser_lab(action="check") runs the 17-check self-test and reports the measured pointer error. hugpy-browser lab-image prints the archival image's location, checksum and install steps; assets/ has the libvirt XML and prep_guest.sh to rebuild the guest from scratch.

Tests

tests/test_engine_aim.py — the engine's own 60+ test suite (wire geometry, grid drawing and parsing, both aim strategies end-to-end against a fake model answering from real pixels, OCR matching, QMP event streams, verified clicks, the lab readiness logic), run against the vendored engine.