hugpy-browser

Screen-driven browser control for bots — no Selenium, no WebDriver, no debugging port. The tools look at the screen and move the real pointer, so a site sees an ordinary user's browser. Measured: pointer error 0 px on VM and desktop; description aims 11/12 on target, 1 px median error (Qwen2.5-VL-7B).

This package is the one place for the browser-perfect work. The engine itself (the browser_* tools: browser_go/look/click/type/move/drag/scroll/wait/read/shot/ locate/surfaces/lab) ships inside abstract-toolserver (>= 0.0.70, a dependency of this package) and runs in any toolserver — on ae that is the 7004_hugpy_toolserver service. What this package adds around it:

Install

pip install hugpy-browser

Quickstart (through a toolserver)

browser_surfaces()                                   # where can I drive?
browser_go(url="https://example.com", on="vm:<name>")
browser_look()                                       # the screen as text items
browser_click(text="Sign in")                        # exact visible words
browser_type(text="me@example.com", into_text="Email")
browser_wait(text="Welcome", timeout=20)

Aim four ways: text= (OCR, exact visible words), target= (vision-model description, confirmed on a magnified crop), cell="J2.D5" (labelled grid, for a bot that reads images itself), or x=, y=. Every click is verified: before/after frames, diff regions, a verdict (changed_at_click | changed_elsewhere | no_change).

For a repeatable environment: browser_lab(action="start") boots an ephemeral clone of the hugpy-browser-lab image (Firefox on a precision test page); browser_lab(action="check") runs the 17-check self-test and reports the measured pointer error. hugpy-browser assets prints the directory with the libvirt XML and guest-prep script to rebuild that image anywhere.

CLI

hugpy-browser where     # the map: engine, service, lab, docs, backups
hugpy-browser guide     # the full operator guide
hugpy-browser numbers   # the measured benchmark results
hugpy-browser assets    # path to precision.html + lab VM assets

Re-measuring

See docs/measure.md and bench/: evalrun.py drives the aim pipelines over the synthetic target set (data/ 233 targets; data2/ row/card targets); vlclient.py points at any OpenAI-style vision endpoint (VL_BASE). Results of record are in docs/final_numbers.md.