abstract_toolserver

The abstract_* ecosystem exposed as an API-callable AI toolset — every tool is a plain Python function turned into a self-describing HTTP endpoint by abstract_flask. A portable tool layer any model (Claude, hugpy, …) can drive over HTTP instead of being bound to one runtime's tool harness.

Part of the hugpy orbit

                    ┌──────────────────────────── hugpy (fleet) ───────────────────────────┐
                    │ central + workers: platform · engine · fleet · server · media · …     │
                    │ OpenAI-compatible /v1 — every local model, incl. B (Qwen3-Coder-Next) │
                    └───────▲───────────────────────▲──────────────────────────▲───────────┘
                            │ inference             │ inference                │ B reductions
   ┌────────────────────────┴──┐   ┌────────────────┴──────────┐   ┌───────────┴───────────────┐
   │ hugpy-station             │   │ hugpy-agent               │   │ abstract-toolserver       │
   │ desktop + headless console│──▶│ agent runtime · TUI ·     │◀─▶│ comms · ledgers · boards ·│
   │ tmux seats per locus      │   │ OpenCode/qwen seats       │   │ exchanges · MCP · b_ask   │
   └────────────┬──────────────┘   └────────────┬──────────────┘   └───────────▲───────────────┘
                │ keeper/codex seats            │ --serve                      │ tools (MCP/HTTP)
   ┌────────────▼──────────────┐   ┌────────────▼──────────────┐               │
   │ abstract-gpt (Codex seat) │   │ abstract-claude serve ────┼───────────────┘
   │ abstract-claude (Claude)  │   │  └ abstract-serve-core    │
   └───────────────────────────┘   └───────────────────────────┘
          everything ships through abstract-pypit → PyPI (+ GitHub)

Run it

pip install abstract_toolserver          # + the extras you want to expose
python -m abstract_toolserver            # HOST/PORT/DEBUG from TOOLSERVER_* env

from abstract_toolserver import get_toolserver_app
app = get_toolserver_app()               # a normal Flask/WSGI app

Self-describing surface

The app auto-mounts introspection endpoints (from abstract_flask):

Call a tool with JSON; unknown keys are pruned to the function signature, and the reply is {"result": ...} (or {"error": ...}). Discover-and-dispatch from a client is already provided by abstract_apis.make_endpoint_call.

curl -s localhost:5000/fs/count_tokens -d '{"text":"hello world"}'
# {"result": 2}
curl -s localhost:5000/db/schema           # {"result": {table: [cols...]}}
curl -s 'localhost:5000/db/query?help=true'

Tool categories

Visual browser (/browser/*)

A browser nobody automated: no WebDriver, no debugging port, no page script. The tools see the screen and move the pointer, so they work on anything that draws a browser window.

browser_surfaces                         where can I drive?  (guests, the desktop, pool guests, the lab)
browser_lab(action="start")              a disposable clone of the controlled lab image -> on="vm:<clone>"
browser_go(url, on=...)                  Ctrl+L, type, Enter, wait until the page stops painting
browser_look(on=...)                     the screen as text: [{id, text, x, y}]  (no model)
browser_click(text="Sign in")            exact visible words (OCR)            | each click walks the
browser_click(id="t12")                  an item of the last look             | pointer there, presses,
browser_click(target="the gear icon")    a description (hugpy's VL model)     | and diffs the screen:
browser_click(cell="C4.B3")              a cell of browser_shot(grid=true)    | changed_at_click |
browser_click(x=412, y=388)              a pixel of the screenshot            | changed_elsewhere | no_change
browser_type(text="...", into_text="Email", submit=true)     browser_key("ctrl+l")
browser_move / browser_drag / browser_scroll / browser_wait(text=...) / browser_read

• Surfaces (on=): vm:<domain> (libvirt guest on this host: virsh screenshot + QMP tablet/keyboard), pool:<name> (a guest on the delegated VM-pool node), screen | monitor:<n> | window:<title> | region:x,y,w,h (the desktop this toolserver runs on — mss + pyautogui — or the node ui is delegated to). Every coordinate is a pixel of the screenshot; origins, HiDPI scale and the tablet range are the surface's business.
• Aiming (aim.py). text= is OCR over two readings (the page at 2x and every text line normalised to dark-on-light) and refuses to guess: absent words are an error, repeated words list their places (nth=). target= asks the VL model: a grounding model (Qwen2.5-VL) answers with a box on a wire image sized for it (sides in multiples of 28, >= 1024 image tokens), a second look on a magnified crop must confirm it — framed around any on-screen text the description names — and tiny results are snapped onto the element. Any other VL model aims by grid-image inference: a labelled grid drawn at wire size (each label in the middle of its cell, where "the nearest label" is the right answer), the chosen cell magnified with most of its neighbours and gridded again, the last cell snapped to the element inside it. A bot that reads images itself uses the same grid: browser_shot(grid=true) -> browser_shot(grid=true, cell="C4") -> browser_click(cell="C4.B3").
• Results are data a bot can branch on: {ok, point: {x, y, method, confidence}, verdict, regions, ...}; a target that is not on the screen is never clicked: {ok: false, error: {code: "not_visible"}, hint}.
• The lab: hugpy-browser-lab is a shut-off libvirt image (Ubuntu GNOME, Firefox with a fresh profile and fixed policies, home page = a precision test page, no NIC). browser_lab boots/destroys disposable clones of it, so a test starts from the same pixels every time. browser_lab(action="check") runs the self-check on a fresh clone (pointer error in pixels, text aims, typing, hover, drag, wheel, keys) and action="check_status" returns the pass/fail table.

Measured 2026-10-06 (233 labelled targets on five mock screens; real Chrome on Xvfb; Firefox in a lab clone): pointer error 0 px on both surface kinds; exact-text aims reach 81 of 82 visible labels with none wrong (the previous exact-text path: 36 of 82, 4 wrong); description aims with Qwen2.5-VL-7B land on the element for 85% of 40 targets in a single look, and on the hardest kind — small icons, checkboxes and buttons told apart only by a row label or card title elsewhere on the screen — for 14 of 14 with the row-framed second look (median error 1 px; a single look: 11 of 14). On the same 12 targets: the 0.0.64 grid 0 of 12, grid-image inference as released 8 of 12 (10 within 8 px), the default pointing path 11 of 12 with a median error of 1 px. Guide: instructions/hugpy/hugpy-browser.

Operating instructions

Tool schemas describe individual calls; the built-in instruction tree documents how calls compose into repeatable workflows:

• GET /tools/toolserver/ — MCP configuration, discovery, runtime-neutral channel comms, and optional runtime-specific wake-up adapters.
• GET /instructions/ae/solcatcher/ — Solcatcher-specific entry point.
• GET /instructions/a-brain/alpha/ — Alpha's capability-channel instructions.
• MCP: instructions_tree, then instructions_read, through ts_call.

Authenticated MCP callers may create a new document with instructions_add at an instructions/... path. Creation is durable and immediately readable, but MCP intentionally exposes no update or delete tool. Those operations remain local to server administration, and built-in documents are immutable.

Safety gates

Backends load lazily, so the server boots on a headless box and a missing backend errors only when its tool is called. Beyond that:

• /db/query — read-only gate: rejects anything that isn't a single SELECT/WITH, blocks stacked statements and data-modifying keywords. Use /db/fetch (identifier-composed, params-not-SQL) as the default read path.
• /sys/run_cmd — disabled unless TOOLSERVER_CMD_ALLOWLIST=ls,grep,… is set; only allowlisted binaries run.
• /fs/read_file · /fs/write_file — local-only; the underlying SSH/remote kwargs are never exposed at the boundary.
• /session/pull — the agent's own MCP bridge (abstract-claude mcp) fills the calling session's identity (session id, user@host, cwd); the toolserver registers machine + session loci, records a pull handoff, and asks the station (HANDOFF_SPAWN_URL) to seat claude --resume <id> --fork-session there. Without a resume-capable station it stays pending (/session/spin retries) — never a silent fresh seat. A pulled session carries its OWN name in the station (name= → tmux seat and session locus; default sess-<id8>).
• default loci — the station service user (vm_mgr) and the host are always in the /loci/pointers distribution: seeded once into the registry, re-merged into the feed even before the table exists (TOOLSERVER_DEFAULT_LOCI, TOOLSERVER_STATION_USER).
• /ui/click_verify — the click→observe→verify loop: locate (text or image template) → click → re-capture → report whether the screen (or a region) changed. The half most tool APIs lack.

Authentication — the operator token is the ONLY gate

Every route requires TOOLSERVER_OPERATOR_TOKEN, sent as X-Operator-Token: <token> or Authorization: Bearer <token> (the MCP bridge sends both). There is no IP allow-list and no loopback bypass: a LAN, WireGuard or 127.0.0.1 caller without the header gets 401 {"error":"unauthorized"} exactly like the public internet (operator ruling 2026-09-29 — the nginx allow 192.168.x/deny all block that used to front toolserver.hugpy.ai was a second, redundant gate and is gone). With the env var unset the server fails closed (every gated route 401s; startup logs an error).

Open by design (they carry their own credential or expose nothing):

TOOLSERVER_REQUIRE_TOKEN (the old opt-in blueprint gate) is deprecated: parsed, logged as ignored, never enforced.

Configuration

Cross-user permits — permit.* and abstract-permit (2026-10-07)

One toolserver node per user; each acts only with its own user's authority. The host node (local only, TOOLSERVER_PERMIT_HUB=1) is the auth delegator: when one user needs something another user owns, the request goes to the hub over a unix socket (the kernel names the caller — no token), and the operator answers deny / once / always (this path) / always (its tree). Nothing is granted up front; "always" answers become standing rules. The owning user's node performs the action, so no Unix permission is ever loosened.

abstract-permit read /srv/other/notes.md --reason "why"   # requester (waits up to --wait s)
abstract-permit watch                                      # operator: prompts as they arrive
abstract-permit rules | revoke ID | log | nodes

Every non-hub node connects out to the hub at start and registers as the node for its uid. Refused outright: paths owned by the requester itself, by root, by a user with no node, and anything under .ssh / .gnupg. Tables: permit_rules, permit_requests. Session keys (hps_) cannot call permit/*.

Login grants (2026-10-08). abstract-permit login <user> --hours N asks to log in AS another account on this host. On approval the requester's node makes an ed25519 key (~/.ssh/permit/grant-N, never leaves the requester), the owner's node adds the public key to the owner's authorized_keys restricted to from="127.0.0.1,::1",restrict,pty,command="… login-shell --grant N", and the request's note carries the exact ssh -i … <user>@127.0.0.1 line. Every login runs through the shim, which records start/end, the ssh client and the command in permit_logins (abstract-permit logins), and everything done that way acts as — and is audited as — the owner. Expiry/revoke removes the key line (owner's node) and the private key (requester's node).

Node identities. Every node has a static node_id (random, created once in <config root>/node.json — ~/.config/abstract-toolserver, or $ABSTRACT_TOOLSERVER_HOME; abstract-toolserver node). It is in the advertisement and /healthz; the hub pins it to the node's uid at first registration and refuses it from any other uid; a retired node_id stays refused (abstract-permit forget-node). A uid may hold several live node_ids (e.g. a user's own node plus a second instance with its own config root); the first one connected acts for the uid, the next takes over when it goes. Requests record requester_node / owner_node: abstract-permit log --node <id>. For the journal, abstract-toolserver node --dropin prints a unit drop-in (LogExtraFields=NODE_ID=…) so one node's log is journalctl NODE_ID=<id>.

Session keys and the router (2026-10-08). Every session registers with its user's node at start (abstract-toolserver session register --hook from a SessionStart hook) and gets a rotating key tsa_… (24 h, 15 min overlap on rotation; the hub stores only its hash). abstract-toolserver session headers prints {"Authorization": "Bearer tsa_…", "X-Session-Id": …}, rotating when close to expiry — usable as an MCP headersHelper. Nodes accept tsa_ keys of their own user (verified by the hub) with the full tool surface, except the console-only authorization decisions (permit_answer, permit_revoke, permit_node_forget). abstract-toolserver router (default 127.0.0.1:7210) is the one public entry point: it routes by session id to that session's node and streams the response; nodes stay on loopback. A request with no session key but the operator token goes to the host node (the hub's node), which checks the token — how a client-side console (hugpy Station) reaches the permit hub and how seat reporters report.

Every user its own node (2026-10-08). abstract-toolserver node install --user U --port P (root) installs the shared template abstract-toolserver-node@.service (the node runs AS the user from the root-owned shared venv), writes ~U/.config/abstract-toolserver/toolserver.env (127.0.0.1:P, own token, own database toolserver_U, peer auth), creates the role/database, enables and starts abstract-toolserver-node@U, then adds the NODE_ID journal drop-in. Idempotent.

serve is never blocked by another instance (operator, 2026-10-08: the collision guard was removed). A second instance of the same user — own port, own database, own config root (ABSTRACT_TOOLSERVER_HOME=<dir> or its own HOME) — has its own node.json, endpoint.json, toolserver.env and session keys; with ABSTRACT_TOOLSERVER_HOME set it advertises only in that dir, and no instance overwrites an advertisement held by another live instance. The consumer hook ensure_endpoint still uses an existing toolserver rather than starting one.

canvas.* — per-locus ◳ design / flow documents (2026-09-03)

The station's ◳ canvas tab (⬚ design = wireframe.v1, ⋔ flow = flow.v1) and every seat share ONE copy per (locus, kind) in the canvas table:

• POST /canvas/get {locus, kind} → {state|null, rev, by, note, updated}
• POST /canvas/put {locus, kind, state, by?, note?, notify?} — whole document, validated fail-closed, stored verbatim; a flow's rev bumps on every changed put; notify=true also posts a [canvas] high-priority request on the locus board (a deliberate hand-off — never for autosave).
• POST /canvas/list {locus?} → which loci hold which kinds (no bodies).

Writes fire on the locus_change bus (table canvas, id = kind) so open drawers reload live. Through abstract-claude mcp these are the Claude Code tools canvas_get / canvas_put / canvas_list.

B on call — b_ask (2026-10-02)

b_ask(question, text= | ledger_locus=,ledger_task= | path=, model=) sends the source plus the question to B, the fleet's local model, and returns {answer, excerpts[], model, source, chars, clipped}. Excerpts are verbatim passages from the source; B is told never to invent content. This is how an agent reads an oversized ledger or file without loading it whole.

• Model: TOOLSERVER_B_MODEL → HANDOFF_POLISH_MODEL → HANDOFF_JUDGE_MODEL → default Qwen3-Coder-Next-GGUF. Transport: HUGPY_BASE + HUGPY_API_KEY, the same fleet JSON chat the roller uses.
• TOOLSERVER_B_MAX_CHARS (48000) caps the source sent; clipped: true says it was cut. TOOLSERVER_B_TIMEOUT (120 s).
• Latency: ~18 s warm on Coder-Next for a 33 KB file; the first call after an idle period includes the model load (~100 s).
• Ledgers are read straight from the DB, so a b_ask over a ledger is never hit by the MCP result governor below.

Comms

comms_ping writes a durable [ping] board row, then (0.0.47+) makes a best-effort live push to the target locus's registered pointer.serve_url (POST <serve>/api/session/message, to=keeper, wait=false, 5 s, never blocks; result.push reports pushed to …). The lookup ignores the registry's archive status. from_/to ids must match [a-z0-9][a-z0-9_-]{0,62} — no colons. kind=message rows are drained into the target Station's mail tab and auto-closed.

Ledgers, handoffs, boards

• Ledgers (ledger_put/get/list/template) are the handoff state of record per locus/task: goal, rulings, decisions, world state, in-flight step, open questions, pointers. A put replaces the whole doc; a sections-only put fails (500).
• Handoffs (handoff_request/pull/claim) hold the seat handoff; the old file-pointer handoffs are retired.
• Boards: todo_update refuses unknown ids (no todo <id>); board-item format is checked per TOOLSERVER_BOARD_FORMAT=warn|enforce.
• instructions_add needs both path and text.

Exchange recording — every operator message is on record (2026-10-06)

exchange_ingest_transcript (rolling.py prime_transcript) turns a Claude Code transcript into exchanges rows, one per operator message:

• A message the operator sends while the agent is working (Claude Code's queued_command attachment, origin.kind = human) is its own row, holding what answered it; pointers.mid_turn = true. Before this, those messages were dropped entirely.
• The incremental floor is per session, not per locus. Parallel sessions on one locus used to push the locus max(ts) past a turn that was still running, and that turn was never ingested. since_ts=0 still re-primes a whole transcript (idempotent per turn uuid).
• Recording still happens at the Stop hook (end of turn). Prompt-first recording (write the prompt before the model sees it) is the open next step.

Rolling state

rolling_reduce.py is deterministic — no model call. It folds primed exchange rows into objective / done / in_progress / blockers / next_steps / key_paths / open_questions / init_prompt (each list capped at 30). Since 0.0.48, tool-failure blockers expire once newer turns arrive, prose blockers drop when the objective changes, and nothing expires on an empty slice. Station's banner shows blockers[0..3] of /api/frontier/state. An optional model polish (HANDOFF_POLISH=1, HANDOFF_POLISH_MODEL, ≤30 s, skipped if busy) and the legacy judge fold (HANDOFF_REDUCER=judge) remain available.

MCP bridge and the result governor

abstract-toolserver-mcp exposes every tool as mcp__toolserver__<name>; ts_call reaches any tool by name, including ones added after the MCP client loaded its tool list. session_message accepts the cross-locus target <locus>:keeper.

Every MCP result over AC_MCP_RESULT_MAX_LINES (200) or AC_MCP_RESULT_MAX_BYTES (16384) is cut to 75 % head + 25 % tail, the full text is archived (exchange_archive_get locus=… source=mcp-result:<tool>:<ts>), and a [truncated: …] marker is appended. AC_MCP_GOVERNOR=0 disables it; AC_MCP_GOVERNOR_EXEMPT lists tools never cut (default: the fs_read_* tools and exchange_archive_get). Prefer b_ask over reading a truncated result.

Three things the bridge does for tools that take many optional arguments, refuse a call or return pictures (since the 2026-10-06 releases):

• Schemas come from signatures. ?help cannot tell "no default" from "default None", so the bridge used to mark every optional-None parameter required. After discovery it now overlays the server's own catalog (POST /mcp?mode=flat): only parameters without a default are required, and annotated types are kept.
• A refusal keeps its reason. A tool that answers 4xx/5xx with {"error": "<why>"} reaches the model as that sentence (isError), not as "HTTP Error 500".
• Images are images. A result value that is an image data URL (image, data_url, ...) or raw base64 under image_b64 / overlay_b64 is lifted into an MCP image content block and replaced in the JSON by a one-line note, instead of reaching the model as a megabyte of base64 the governor then cuts. AC_MCP_IMAGES=0 disables it.

Deployment on a host

One Flask service per host: unit 7004_hugpy_toolserver (gunicorn as vm_mgr, 127.0.0.1:7004) running an editable install of the dev tree, so a service restart picks up source edits without a publish. Discovery writes endpoint.json. Token resolution order: HUGPY_TOOLSERVER_TOKEN → TOOLSERVER_OPERATOR_TOKEN → TOOLSERVER_TOKEN → HUGPY_OPERATOR_TOKEN → STATION_CONSOLE_TOOLSERVER_TOKEN → env files. Auth fails closed when no token is set; /ch/ channels and per-client token paths are exempt; the /vms/ page uses cookie auth.

Attention-worthy

• The MCP tool list is fixed when a client session starts: a new tool needs a service restart and a new session to appear as mcp__toolserver__* (use ts_call meanwhile).
• ledger_get returns the doc twice (doc + sections), so a 17 KB ledger trips the 16 KB governor — read big ledgers through b_ask.
• The roller's background exchange_ingest_transcript logs no transcript at (none found) when a locus has no transcript; harmless.
• Fleet usage.prompt_tokens can under-report on the hugpy route.