aibrowsertoolkit

An HTTP server that lets an AI agent drive a real browser — Chrome or Edge — by sending JSON. The server process is the loop: it opens the chosen browser with a persistent profile, stays up waiting for commands, and only stops when you send shutdown.

Agents: read guidelines/toolkit-workflow.md before driving anything. Not "if your site looks tricky", not "if there is a playbook for it" — always. Most sites have no playbook, and that file is the whole answer for them. Then, only if one exists, add the site playbook from guidelines/.

Skipping it costs you the things that are hard to guess: that find hands back refs you act on directly, that a click already reports what changed so you need not re-read the page, and that ops can be sent in batches.

abt serve  ──starts──>  FastAPI :8765  ──owns──>  Chrome/Edge (persistent profile)
                             ^
   abt / curl / your agent ──┘   POST /command  |  POST /commands

Install

Needs Python 3.11+ and Google Chrome or Microsoft Edge installed. The matching chromedriver/msedgedriver is resolved automatically by Selenium Manager — nothing to download by hand.

git clone https://github.com/skssmd/Ai-Browser-Toolkit
cd Ai-Browser-Toolkit

python -m venv .venv
.venv/Scripts/python -m pip install -e ".[dev]"     # Windows
# .venv/bin/python -m pip install -e ".[dev]"       # macOS / Linux

Check it worked:

.venv/Scripts/python -m pytest -q     # 485 tests, needs Chrome

The abt command lands in the venv. Either activate it (.venv\Scripts\activate / source .venv/bin/activate) so abt is on your PATH, or call it explicitly as .venv/Scripts/python -m abt.cli … everywhere below.

Run

Start the server as a separate process. abt serve is the command loop — it listens and never returns on its own. Run it in the foreground and it holds that terminal until you shut it down, so an agent or script that launches it inline will hang there forever. Background it, or give it its own terminal.

abt up is the shortest safe way, and the one to reach for from an agent:

abt up            # start a server if none is running, then return

It returns in seconds and never blocks. On Windows it asks WMI (falling back to Task Scheduler) to do the spawning, so the server's parent is a Windows service rather than your shell — which means it sits outside your caller's job object. That distinction is the whole point: a harness that waits on its job will hang on a plain background launch no matter how you redirect stdio, and CREATE_BREAKAWAY_FROM_JOB fails silently when the job forbids breakaway. On POSIX it double-forks with setsid.

The server starts with no browser. It listens in about a second instead of waiting up to two minutes for Chrome on the persistent profile. Ask for a browser when you want one:

curl -X POST http://127.0.0.1:8765/browser/start
curl http://127.0.0.1:8765/browser     # running? on what config?
curl http://127.0.0.1:8765/health      # is the *server* up (never touches the driver)

Pass --start-browser to abt serve for the old eager behaviour.

When the browser dies, the server survives it. Every page command returns browser_dead naming the remedy, and browser_restart gives you a fresh browser on the same profile without losing the server or the session log. You stay logged in; tabs and refs do not survive.

The safe way — use the start script. It does the whole dance for you and is the only thing an agent or CI job should call:

./start-server.sh              # Linux / macOS / Git Bash / WSL
start-server.bat               # Windows cmd

It no-ops if a server is already running, creates .venv and installs dependencies only when they are missing, launches the server detached with its output redirected to server.log / server.err, then polls /health and exits 0 once the server answers (about a second now that no browser is launched at startup). Exit 1 means it never came up — it prints the tail of server.err; exit 2 means dependencies failed.

./start-server.sh --status     # only report up/down, start nothing (exit 0 = up)
./start-server.sh --no-wait    # launch and return immediately, skip polling
./start-server.sh --browser edge --port 9000 --headless   # extra flags go to `abt serve`

A server on a non-default port writes server-<port>.log instead, so starting one never clobbers the logs of the server on 8765.

By hand, if you prefer. Detaching is not enough on its own — redirect the output too, or the launching process keeps waiting on the inherited pipe:

# Linux / macOS — background it and keep the log
nohup abt serve --browser chrome > server.log 2>&1 &

# Windows PowerShell
Start-Process .venv\Scripts\abt.exe `
  -ArgumentList "serve","--browser","chrome" `
  -RedirectStandardOutput server.log -RedirectStandardError server.err

# or just give it its own terminal and leave it there
abt serve --browser chrome

It is ready when /status answers — poll rather than sleeping a fixed amount, since first launch has to start Chrome:

until curl -s -m 3 localhost:8765/status > /dev/null; do sleep 1; done

Options:

abt serve --headless                 # no window
abt serve --port 9000                # another port
abt serve --profile ~/work-profile   # another profile
abt serve --no-log --no-diff         # leanest responses
abt serve --settle-timeout 10        # give a slow SPA longer to render
abt serve --no-frames                # stop reading inside iframes

Stop it with abt shutdown (or POST {"op":"shutdown"}), which closes Chrome and exits the process. Ctrl+C works too if it has its own terminal.

Because the server owns exactly one browser, run one server per browser — use --port and --profile together to run more than one side by side.

The profile directory persists cookies and logins across restarts, and is isolated from your everyday Chrome. Log in by hand once in the visible window and the session sticks around — including across a restart to pick up new code.

Start at login

The toolkit is most useful already up: an agent that has to start a server first pays for the start, and one that starts it wrongly wedges itself.

abt autostart install --browser chrome --dry-run   # read it first
abt autostart install --browser chrome
abt autostart status
abt autostart uninstall

Platform	What it writes
Windows	a Task Scheduler logon task
macOS	~/Library/LaunchAgents/com.aibrowsertoolkit.server.plist
Linux	~/.config/systemd/user/abt-server.service

User-level, never elevated, and opt in — nothing is installed unless you ask. It runs in the account that owns the browser profile, because a system service would run as somebody else and find none of your logins.

No browser is launched at login. The entry runs abt serve without --start-browser, so logging in costs about a second and the browser arrives when you ask for it with browser_start. This is the whole reason the feature is sane: against the old eager behaviour it would have opened Chrome on the persistent profile at every logon.

--browser is required rather than prompted, and every path is resolved to an absolute one at install time. A logon entry has neither a terminal to answer a prompt nor a working directory to resolve against — a relative ./profile would land in C:\Windows\System32 and quietly build a second, empty profile there.

On Linux a user unit stops at logout. For a server that outlives the session:

loginctl enable-linger $USER

Send commands

One command:

curl -s localhost:8765/command -H 'content-type: application/json' \
  -d '{"op":"goto","url":"https://example.com"}'

A list, run in order:

curl -s localhost:8765/commands -H 'content-type: application/json' -d '[
  {"op":"goto","url":"https://example.com"},
  {"op":"find","css":"a"},
  {"op":"click","ref":"el_0"},
  {"op":"get_text","css":"h1"}
]'

Every response is {"ok": true, "result": ...} or {"ok": false, "error": {"type", "message", "op_index"}}.

Batches stop at the first failure by default. Send {"commands": [...], "continue_on_error": true} to run them all regardless.

Messenger

Driving messenger.com through the generic ops takes a dozen round trips and the order matters — a stale draft glued to your text, an Enter that fires before the upload finished. That sequence is packaged as its own endpoints:

curl -s localhost:8765/messenger/sendmessage -H 'content-type: application/json' -d '{
  "thread_url": "https://www.messenger.com/t/927345869967156/",
  "message": "@Yaleed @Samin here is the capture",
  "mentions": ["Yaleed", "Samin"],
  "attachments": ["C:/shots/page.png", "https://example.com/logo.png"],
  "reply_to": "Step 1/4 DONE"
}'

mentions are real @-mentions: each name must appear in message as @<name>, and that spot is typed through Messenger's suggestion popup, so @Yaleed lands as @Yaleed Haque. attachments take local paths or http(s) links, which are downloaded first. reply_to is a substring of the message you are answering, or an index into the thread.

Every failure raises before Enter is pressed — an attachment that never staged, a mention with no suggestion — so a bad send stays a draft instead of going out wrong.

POST /messenger/sendmessage/async answers immediately with a job_id and does the work in a new tab that it closes afterwards, leaving your current tab untouched. Poll GET /messenger/jobs/{id}.

Reading:

curl -s 'localhost:8765/messenger/threads?url=https://www.messenger.com/'
curl -s 'localhost:8765/messenger/messages?thread_url=…&since_last=true'

/messenger/threads gives each thread's name, preview, time, and URL. /messenger/messages parses rows into {sender, time, text}; since_last=true returns only what arrived since your last read of that thread — matched by content, not position, because Messenger trims the top of a long thread as it grows.

Full details and the traps behind them: guidelines/messenger.com/messenger.md.

Console and network

The DOM cannot tell you why a request failed. These can:

curl -s localhost:8765/command -d '{"op":"read_network","failures_only":true}'
curl -s localhost:8765/command -d '{"op":"read_console","levels":["error"]}'

read_console captures from document start, so a reload hands back what the page logged while loading — uncaught errors and unhandled rejections included. read_network returns each request with its status, duration, and size; failures_only keeps the 4xx/5xx and anything the browser would not disclose. Both take a pattern regex.

Two ways to read a page

find returns element shells — each match's own tag and attributes, with all children and text stripped. This is how an agent surveys page structure without paying for the content:

{"op": "find", "css": ".card"}
→ {"count": 3, "matches": [
     {"ref": "el_0", "html": "<div class=\"card\" id=\"p1\"></div>", "visible": true}
   ]}

find_full (or {"mode": "full"}) returns the same matches with everything inside:

{"op": "find_full", "css": ".card"}
→ {"matches": [{"ref": "el_0", "html": "<div class=\"card\" id=\"p1\"><h2>Cheap Widget</h2>…"}]}

Element refs

Every find match gets a ref (el_0, el_1, …). Act on it directly instead of writing another selector:

{"op": "find",  "css": "button.buy"}
{"op": "click", "ref": "el_0"}

Refs die when the tab navigates or the element leaves the DOM. Using a dead ref returns stale_ref telling you to search again — it never quietly hits a different element. Each tab keeps its own refs.

Numbering does not restart after a navigation: the counter runs for the tab's life, so el_0 never means two different elements. That is what makes the guarantee above true — if numbering restarted, the new page's el_0 would answer to a handle you were still holding for the old one. Expect the numbers to climb over a long session.

Targeting

near — one selector, many matches. A documents table gives every row an Edit button, so text: "Edit" matches a dozen and index is a guess:

{"op": "click", "text": "Edit", "near": "Medication"}

The match sharing the smallest container with that text wins — its row on a table, its section on a card. Size rather than depth, because a control sitting directly in <body> has every string on the page as a zero-depth ancestor and would otherwise beat the right answer.

near qualifies a selector, so it combines with css, xpath or text, and is refused on a ref — a ref already names one element. When nothing is near it, the error lists what is.

Ops that touch an element take exactly one of ref, css, xpath, or text (exact visible text). When a selector matches several elements, acting ops use the first unless you pass index.

Diffs: what the last command changed

Every interactive op (click input select hover scroll press wait_for run_js) and every navigation op (goto back forward reload) reports what it changed as a dom_diff key on its response. The page is snapshotted before and after; the two snapshots difflib into what appeared and what disappeared. This is the feedback loop — you find out what a command did without re-fetching the page.

There are three tracks, and they answer different questions.

The text track — always on

What appeared on screen. Every element's own text plus every form control's live value, collected in document order as separate entries. One element, one entry: two adjacent labels stay two strings and never merge into a blob, so you can tell which text belongs to which thing without any markup being present.

{"op": "click", "css": "#products"}
→ {"clicked": "css='#products'", "forced": false, …,
   "dom_diff": {
     "url_before": "https://shop.example/", "url_after": "https://shop.example/",
     "text": {"added": ["Widgets", "Gadgets", "3 items"],
              "removed_count": 1,
              "truncated": false}
   }}

This carries no tags, no classes, and no attributes, so it stays small enough to be unconditional — there is no budget to set and nothing to tune. What it costs you is state that has no visible text: a class flip, an aria-expanded toggle, or a data- attribute produces an empty text diff.

Removals are counted, not listed. Additions are what you act on — the new options, the result, the error. What left is usually the page you were already looking at, and on a page that replaces its body the removals are the entire old document for no benefit. So you get removed_count for free, and the strings themselves only when you ask:

{"op": "click", "css": "#products", "include_removed": true}
→ …, "text": {"added": […], "removed": ["Loading…"], "removed_count": 1, …}

The count is there so you can tell when asking is worth it: removed_count: 1 after a click is a spinner disappearing, removed_count: 340 means the page rewrote itself.

When the page navigates, you get the page. A diff against a document that no longer exists is meaningless, but the question behind it — what am I looking at now? — still has an answer. So added becomes the destination's full text instead of going empty, and you skip the separate read you would otherwise need.

The page is waited for first. driver.get returns when the document has loaded, which on a single-page app is the moment a spinner mounts and nothing else has rendered — so without this the diff reported Loading dashboard… / Please wait while we process your request as though that were the page, and an agent learned to distrust it and re-read the body every time. Navigation now settles on two signals before snapshotting: no request in flight (a slow fetch holds the DOM perfectly still, so a DOM-only check would call the spinner "done") and a DOM that has stopped changing (which catches a render that owes nothing to the network). A page that never stops — a poller, a clock — costs --settle-timeout and then proceeds, because a late diff beats no diff.

{"op": "goto", "url": "https://shop.example/cart"}
→ {"url": "…/cart", "title": "Your cart",
   "dom_diff": {"navigation": true,
                "note": "text is the full page you landed on, not a diff",
                "text": {"added": ["Your cart", "2 items", "Total: $42", "Checkout"],
                         "removed_count": 18, "truncated": false}}}

This covers goto back forward reload and any interactive op that redirected — a click that leaves the page returns the page it arrived at. The element track is skipped in this case, since a unified diff of two unrelated documents is noise at any budget. "diff": false turns it off per command.

Only rendered text counts. An element with display: none has not appeared yet, so a hover that reveals a menu shows up as its items being added — which is usually exactly the event you were waiting for. Password field values are never captured, since diffs are written to the session log.

The element track — opt in

Which element the text belongs to. Pass element_diff: true for the line-per-element unified diff: tag, id, classes, sorted attributes, own text. This is the one to reach for when the change was an attribute with no text at all, or when you need a selector for something the text track just told you appeared.

{"op": "click", "css": "#insert-menu", "element_diff": true}
→ {…, "dom_diff": {
     "text": {"added": ["Chart", "Pivot table", …], "removed_count": 0, "truncated": false},
     "elements": {"added": 2, "removed": 1, "truncated": false,
                  "diff": "@@ -1 +1 @@\n-div#insert-menu [aria-expanded=\"false\"] …"}
   }}

added/removed are exact counts; diff is the unified diff; truncated means it outgrew the token budget — the counts are still right. Budget it with diff_max_tokens per command or --diff-max-tokens on the server. Passing diff_max_tokens implies element_diff: true, since budgeting something you never asked for is a typo.

{"op": "click", "css": "#menu", "diff": false}              // no diff at all
{"op": "click", "css": "#menu", "include_removed": true}    // list what left too
{"op": "click", "css": "#menu", "element_diff": true}       // both tracks
{"op": "click", "css": "#menu", "diff_max_tokens": 20000}   // both, generous budget
{"op": "click", "css": "#menu", "actionable": false}        // skip refs and roles

The actionable track — on by default

Which of what appeared you can actually click. The text track hands you strings. A string is not addressable, so acting on one used to mean a find to turn it back into an element. This closes that gap: the interactive elements among the additions come back with their role, their name, and a ref.

{"op": "click", "css": "#insert-menu"}
→ {…, "dom_diff": {
     "text": {"added": ["Chart", "Pivot table", "Macro"], "removed_count": 0},
     "actionable": {"added": [
       {"ref": "el_7", "role": "menuitem", "name": "Chart"},
       {"ref": "el_8", "role": "menuitem", "name": "Pivot table"},
       {"ref": "el_9", "role": "button", "name": "Macro", "disabled": true}
     ], "truncated": false}
   }}

{"op": "click", "ref": "el_8"}     ← act on it directly; no find in between

role is the ARIA role, explicit if the element declares one and implicit from the tag otherwise. name is the accessible name — aria-label, then aria-labelledby, then the associated <label>, then the element's own text. disabled appears only when the control is disabled, so you can tell "it showed up but you cannot use it yet" from "it is ready".

Text is the anchor. Every entry's name is a string the text track reported in the same response, so the two always line up. A control with no accessible name is dropped entirely rather than handed back as a nameless ref — an entry you cannot tie to something you have read is noise.

Repeated names arrive qualified. A name identifies a control only while it is unique — a table whose every row carries an Edit button gives you N refs and nothing to choose between them. When a name appears more than once in one diff, each entry gains near: the nearest text that is not the control's own.

"actionable": {"added": [
  {"ref": "el_9",  "role": "button", "name": "Edit", "near": "Medication"},
  {"ref": "el_10", "role": "button", "name": "Edit", "near": "Passport"},
  {"ref": "el_11", "role": "button", "name": "Save Changes"}
]}

The walk climbs ancestors until something else has something to say, which on a row is its first cell and on a card is its heading — it knows nothing about rows or cards. Save Changes gets nothing because nothing else is called that.

This is conditional by construction: no repeated name means no extra work and a response identical to before. It exists because the alternative is run_js DOM-walking to match a button to its row, which is what once clicked the wrong row's Edit and reported success.

It does not run after a navigation. On a new document every control is new, so the diff would degenerate into an inventory of the whole page — which the text track already returned in full, and which would burn a ref on every control to say it. Use find when you land somewhere; use this when something appeared.

Uploads are the one exception to "must be rendered". The standard pattern hides the real <input type=file> behind a custom control that validates or resizes, so the element you must send a path to is never the one on screen. File inputs are therefore reported even when invisible, with role: "file", a name taken from their <label> (falling back to name or id), and multiple: true when they accept more than one. input writes to them hidden:

{"op": "input", "css": "#upload", "value": "C:/shots/page.png"}

What it costs. Unlike the text track this one is not free: about a quarter again on top of a diffed op, and more when controls actually appear. Turn it off per command with "actionable": false for the steps in a batch whose new controls you will never click.

Frames — on by default

A frame is a separate document, and everything reads straight through it. All three tracks, plus find, get_text and every selector, cover the frames a page embeds as well as the page itself. Refs from inside one act normally; the toolkit remembers which document each came from and goes back there.

This exists because the alternative is silence. On linkedin.com/login, Google draws Continue with Google inside a frame from accounts.google.com. Before this, the button was plainly on screen and:

{"op": "find", "text": "Continue with Google"}   → {"count": 0, "matches": []}
{"op": "get_text"}                               → "…Sign in with Microsoft\nSign in with Apple…"

No error, no warning, no truncation notice — just a confident answer with the page's main control missing from it. An error is something an agent reacts to; an empty answer is something it believes. The same blindness covers Stripe card fields, CAPTCHAs, embedded editors, and most OAuth widgets.

What it costs. Nothing on a page with no frames — the snapshot script already walks the document, so it reports the frames it found in the call that was happening anyway, and a frameless answer ends there without one extra request. Measured on a frameless page: 9.9 ms before, 9.2 ms after. A page with one frame pays about +60 ms per snapshot for content it previously could not see at all. At most 8 frames and 2 levels deep, and frames too small to see or click are skipped — including the 0×0 preload sign-in widgets mount beside their real button.

A faster version of this entered frames by number instead of by element, which saved about 32 ms a frame. It was wrong. A page has two orderings of its frames — document order, and the window.frames order the WebDriver spec says a number indexes — and on linkedin.com/login they are exactly reversed. It skipped Google's 0×0 boot frame, entered it anyway, and reported its contents as the page. See frames.enter if you are ever tempted to speed this up.

Turn it off with --no-frames, or tune with --max-frames / --max-frame-depth.

Shadow roots — counted, not walked

Frames are folded into the tracks because a page that has them almost always has content in them. Shadow roots are the opposite trade, so they are opt-in: most pages have no author roots at all, and the ones that do keep component internals there. On the LinkedIn page that prompted this there was one root and nothing in it — walking it every snapshot would have cost the whole session and bought nothing.

So the snapshot counts hosts without looking inside. That is one property read on a walk already happening (measured: no change from ~11 ms), and it turns an empty diff from a silence into a signpost:

{"op": "click", "css": "#next"}
→ {"dom_diff": {"text": {"added": []},
                "shadow": {"hosts": 2, "note": "not walked; …"}}}

{"op": "find", "css": "#resume"}
→ {"count": 0, "shadow_hosts": 2, "note": "… retry with \"shadow\": true"}

Both appear only when hosts exist and the result is otherwise silent, so an ordinary diff on an ordinary page carries neither.

To look inside, ask:

{"op": "find", "css": "input[type=file]", "shadow": true}
→ {"count": 1, "matches": [{"ref": "el_12", "html": "…", "shadow": true}]}

Refs from a shadow root click like any other. css and text only — the walk is querySelectorAll on each root, which is the only way across the boundary and does not speak xpath.

Note that get_text already reports open shadow content, because rendered text follows the composed tree. So discovery is free and shadow: true is only needed to turn something you can read into something you can act on.

The limit: mode: "closed" makes .shadowRoot null. No JavaScript can read such a root or prove it exists, so nothing here can either. "Not found" means nothing reachable has it — see the search ladder in guidelines/toolkit-workflow.md.

The manual check

{"op": "diff"} diffs the current page against the baseline — the state after the last command that touched the page. Use it to catch async SPA updates that land after the command that triggered them. Because you asked for it explicitly, it returns everything by default: both tracks, removals listed.

{"op": "diff"}
→ {"baseline": "present", "tab_id": "tab_0", "url_before": "…", "url_after": "…",
   "text": {"added": ["Order confirmed"], "removed": ["Placing order…"],
            "removed_count": 1, "truncated": false},
   "elements": {"added": 3, "removed": 2, "truncated": false, "diff": "…"}}

{"op": "diff", "reset": true}             → baseline is now the current page
{"op": "diff", "element_diff": false}     → text alone
{"op": "diff", "include_removed": false}  → count removals instead of listing them
{"op": "diff", "max_tokens": 500}         → smaller element-diff budget

Two behaviours worth knowing

Every DOM-touching op re-baselines after itself, so the op that caused a change also consumes it. A manual {"op": "diff"} straight after a click comes back empty — the click already reported that change. Each tab keeps its own baseline; diff on a fresh tab sets one instead of failing.

Diffing costs two snapshots per op — roughly 180 ms on a heavy SPA like Google Sheets, far less on ordinary pages. Interactively that is invisible; across a 60-command batch it is about 11 seconds. Use "diff": false on the commands in a batch whose effects you do not care about, --no-diff when running a script you have already debugged, and "diff": true to force one back on when the server default is off.

Ops

Group	Ops
Navigate	goto back forward reload current_url
Read	get_html get_text find find_full screenshot
Interact	click input select hover scroll wait_for press
Tabs	tab_new tab_list tab_switch tab_close
Control	run_js diff status shutdown
Browser lifecycle	browser_start browser_stop browser_restart browser_status

select drives native <select> elements via by_text, value, or option_index. Custom dropdown navs that only open on mouseover are hover then click.

press: keys, named keys, and chords

press sends a single character, a named key, or a modifier chord to whatever has focus (or to a target when one is given):

{"op": "press", "key": "H"}
{"op": "press", "key": "Enter"}
{"op": "press", "key": "ctrl+v"}        // paste from the clipboard
{"op": "press", "key": "ctrl+alt+1"}    // Google Docs: Heading 1
{"op": "press", "key": "shift+enter"}

Modifiers combine with + and are named ctrl/control, shift, alt/option, and meta/command/cmd/windows. A chord is one or more modifiers plus a final key (a single character or a named key). Unknown keys and modifiers are rejected up front with invalid_op.

Clicking past an overlay, and opening in a new tab

Ad networks park a transparent div at max z-index over the whole page, so the first click anywhere is swallowed. click reports that as not_interactable and names the intercepting element. Two opt-ins deal with it:

{"op": "click", "css": "a.result", "force": true}     → dispatches a JS click
{"op": "click", "css": "a.result", "new_tab": true}   → opens the href beside you

force only defeats occlusion. Hidden and disabled targets are ruled out before it engages, so it never fakes input a user could not have given. The result says "forced": true only when the fallback actually fired.

Without force, click hit-tests before it dispatches: whatever sits at the target's centre must be the target, an ancestor of it, or a descendant. Selenium judges an element by its own state and never asks what is painted over it, so without this check a click could be swallowed by an overlay while the op reported ok: true — the worst answer an agent can be handed. The error names the element that would have received the click instead:

not_interactable: css='a.result' is covered by div#promo-overlay, which would
receive the click instead. Pass force:true to dispatch it anyway, or
new_tab:true if the target is a link

new_tab reads the target's href and opens it in a fresh tab, returning its tab_id. Nothing is clicked, so an overlay is irrelevant — which makes it the cleaner answer of the two when the target is a link. Add "activate": false to open it in the background. The two flags are mutually exclusive.

abt click --css "a.result" --force
abt click --css "a.result" --new-tab --background

GET /ops lists them live; GET /status reports the current URL, open tabs, and live ref count without waiting for a running command to finish.

Errors

A closed set, so you can branch on error.type instead of parsing prose:

invalid_op element_not_found stale_ref not_interactable not_a_select timeout navigation_failed js_error last_tab tab_not_found browser_dead

A failed page load is caught even though Chrome reports success for it — Chrome renders its own error page, and goto raises navigation_failed rather than letting an agent read that page as if it were the site.

Session logs

Every command is recorded to ./logs/<session-id>/events.jsonl — one line per command, holding what ran, what came back, which tab it ran in, and which site was loaded at the time. One session per server run.

JSONL rather than a JSON array so a killed server still leaves every line up to the kill readable. Long strings are truncated with a marker, so a get_html of a big page doesn't bloat the log. Logging can never fail a command: recorder errors are swallowed.

abt serve --log-dir ./logs   # default
abt serve --no-log           # off

Frames: the visual audit trail

Alongside each command the server stores a screenshot of the page it produced. An agent that worked unattended for an hour leaves a log saying it clicked Save; the frame beside that line is how you check it clicked Save on the right page.

Frames are captured for the ops that change what is on screen — goto, click, input, press, select, hover, scroll, the tab ops — plus every failure, whatever the op was. A find or get_text changes nothing, so it gets none. They are JPEGs, downscaled to 1280px at quality 60, and consecutive identical frames are stored once, which puts a long session in the single-digit megabytes.

When the command acted on an element, the frame records where that element sat as fractions of the viewport, and the viewer draws a box on it — so you see what was clicked, not just that something was. A click that navigated away leaves no box: the element is gone, and a box would be pointing at a page that no longer exists.

abt serve --no-shots            # off
abt serve --shot-quality 40     # smaller files
abt serve --shot-width 900      # smaller still
abt serve --shots-max-mb 50     # stop capturing past this much, per session

Frames live in logs/<session-id>/shots/ and are served at GET /logs/{session_id}/shots/{name}, where name is the shot field on the event. They contain whatever was on screen, including logged-in accounts and private messages — treat logs/ as secret.

Browse them in the browser at http://127.0.0.1:8765/viewer — a self-contained page that lists sessions and sites down the side. It opens on a timeline: one row per command, its frame on the left, what the agent did and what came back on the right, failures tinted red, with a header counting commands, errors, elapsed time and sites. Click any frame to enlarge it and step through the session with the arrow keys. Switch the first dropdown to raw log for the older request/response view. Filter by tab, site, op, errors only, or frames only.

Or over the API:

Endpoint	Returns
GET /logs	Every session, newest first, with event and error counts
GET /logs/sites	Every site seen across all sessions
GET /logs/{session_id}	That session's events
GET /logs/{session_id}/shots/{name}	One captured frame, as named by an event's shot

/logs/{session_id} accepts ?site=, ?tab=, ?op=, and ?errors_only=true, which combine.

abt logs                              # list sessions
abt logs --sites                      # list sites
abt logs 20260803-061843              # one session
abt logs 20260803-061843 --errors     # just the failures
abt logs 20260803-061843 --site example.com --tab tab_1

An event looks like this:

{"seq": 4, "at": "2026-08-03T05:18:49Z", "session_id": "20260803-061843",
 "tab_id": "tab_0", "site": "example.com", "url": "https://example.com/",
 "op": "click", "ok": false, "error_type": "element_not_found",
 "duration_ms": 5190.7,
 "request": {"op": "click", "css": "#does-not-exist"},
 "response": {"type": "element_not_found", "message": "nothing matched …"},
 "shot": "00004.jpg",
 "shot_box": {"x": 0.2, "y": 0.15, "w": 0.6, "h": 0.04}}

shot names a file under logs/<session-id>/shots/; shot_box is where the command's target sat, as fractions of the frame. Both are absent when there was nothing to capture or nothing to point at.

CLI

abt is a thin client over the same API; anything it does, curl can do.

abt goto https://example.com
abt find "a.product" --limit 20
abt find "a.product" --full
abt click --ref el_0
abt input "hello" --css "#search"
abt exec '{"op":"select","css":"#size","by_text":"Large"}'
abt exec-batch steps.json --continue-on-error
abt tabs
abt status
abt diff
abt diff --reset
abt logs
abt shutdown

The Messenger endpoints have their own group:

abt messenger threads --url https://www.messenger.com/
abt messenger read -t https://www.messenger.com/t/<id>/ --new
abt messenger send "@Yaleed here it is" -t <thread-url> -m Yaleed -a C:/shots/page.png
abt messenger send "step 2 done" -t <thread-url> --async
abt messenger jobs <job-id>

MCP

abt mcp serves the ops as typed tools over stdio, for any MCP client — Claude Code, Claude Desktop, Cursor, VS Code:

{"mcpServers": {"browser": {"command": "abt", "args": ["mcp"]}}}

It is a proxy, not a second server. abt serve still owns the browser, so the window, its tabs and its logins outlive your editor — start it first. The shim owns nothing and can come and go.

Why bother, when curl already works: driving the toolkit through a shell means the model writes each request as a shell command, and it gets them wrong. One 66-command session against a live site produced five schema errors, every one a guessed parameter name — label on a select, diff on a get_text, a run_js script sent as an object. Typed schemas make those unrepresentable, and the JSON never meets a shell, which on Windows is its own source of quoting failures.

Thirteen tools rather than one per op, since every schema sits in the model's context for the whole session. browser_batch sends a whole sequence in one call, and browser_command passes through any raw op the named tools miss.

Tests

.venv/Scripts/python -m pytest

The suite drives a real headless Chrome against static fixture pages served from a local port — no network, deterministic. Around seven minutes for the full run.

.venv/Scripts/python -m pytest --engine playwright   # the same 485 against Playwright

Two engines, one suite. Playwright is the default; --engine selenium runs every assertion against the Selenium backend instead. Both pass. The flag exists because "a caller cannot tell which engine is underneath" is a claim worth checking by running the tests rather than by reading the diff.

tests/test_engine.py needs no browser and runs in under a second. It guards the driver seam: that nothing outside engine.py and browser.py imports Selenium directly, and that the key table stays derived from the driver rather than typed out by hand. Both are properties no single diff makes visible.

Licence

Apache License 2.0 — © the Ai-Browser-Toolkit contributors.

You may use, modify, and redistribute this code, including commercially. In return the licence asks for three things, and they are not optional:

1. Credit the project. Mention this repository in your own README, or cite it, with a link to https://github.com/skssmd/Ai-Browser-Toolkit.
2. Ship the licence. Keep LICENSE and the copyright and attribution notices with any copy or derivative work.
3. Say what you changed. Mark modified files as modified.

Taking the code without the credit is not "borrowing" — it is using it outside the terms that made it available to you.