hugpy-router

One OpenAI-compatible address in front of every box's hugpy-wrapper front door. Per call it picks the box, proxies the call — streaming, uploads and binary answers pass through as they arrive — and when a box refuses before the first byte it tries the next one. Every call is one row in the router's own log, linked to the box's row.

client ──> hugpy-router ──> hugpy-wrapper (box A) ──> engine
                 └────────> hugpy-wrapper (box B) …   (bounce on refusal)

How a box is chosen

1. Pin — {"alloc": {"worker": "<box>"}} in the body: that box or a refusal. Never rerouted.
2. Eligible — online, the call's pool, not blocked, not already tried for this call.
3. Serving first — a box where the model is loaded, healthy, not loading, and running the same settings (model@options — the options the router last sent that box for that model). A loaded model with other settings is not a match: the box would reload it. Idle before busy, then most recently used.
4. Load — boxes whose catalog holds the model, most free GPU first, then least recently picked.
5. Refusal — none qualifies: 503 no_box with every box's reason.

A box that answers load_error, insufficient_resources, profile_materializing, model not found, 502/503/504, or is unreachable — before the first byte — is skipped and the next one tried (at most 3 boxes per call). Other errors (e.g. a 400) go back to the client exactly as the box sent them.

Run

pip install hugpy-router
hugpy-router boxes add --name ae --url http://127.0.0.1:40717 --key-file /etc/hugpy/wrapper.env \
    [--store /var/lib/hugpy-wrapper/fitevict.sqlite3]
hugpy-router serve --port <n|auto> [--host 127.0.0.1]

No port is assumed. --store (optional): when the box's store file is readable on this machine (hugpy-wrapper --system makes it readable by group hugpy), the router reads the box's state straight from it, re-reading only when SQLite reports a commit. Otherwise it follows the box with a long-poll on GET /v1/telemetry/state?wait=30&since=<n> (hugpy-wrapper ≥ 0.1.52). Either way the box's /health decides online / offline.

The boxes file (HUGPY_ROUTER_BOXES, default <config dir>/router/boxes.json, 0600) holds each box's key; clients never see box keys.

Routes

Responses carry X-Hugpy-Box and X-Hugpy-Router-Request-Id; the box's own call row has that id as its parent.

Keys

The same scheme as hugpy-wrapper (hpk_<id>_<secret>, Authorization: Bearer), with the router's own settings: HUGPY_ROUTER_KEY (its admin key), HUGPY_ROUTER_REQUIRE_KEY (outside by default — no key from loopback and private networks), HUGPY_ROUTER_TRUSTED_NETS, HUGPY_ROUTER_TRUSTED_PROXIES (forwarded requests count as outside unless their proxy is listed). Admin routes need an admin key except from loopback.

hugpy-router keys create --name laptop [--scope use|admin] --router-port <n>

Store

HUGPY_ROUTER_HOME, else systemd's state directory, else <data dir>/router: one SQLite file (hugpy-wrapper's store machinery — WAL, migrations, integrity check): route_log (one row per call: decision, attempts, box, timings) and api_keys.

Layout

hugpy_router/
  core/           types + decide (pure, stdlib only)
  links/          box links (local store file / remote long-poll) + the boxes file
  serve/          aiohttp app: proxy (hot path), management API
  observability/  the router's store + schema/
  cli/            hugpy-router serve / boxes / keys
  imports/        constants (every env name), functions, classes

0.0.1 is phase 1 of the design (one or many boxes, local or remote links). Next: an installer on the hugpy layout (user hugpy-router, /etc/hugpy/router.env, unit), and hugpy-core configuring the router (verdicts, pools, blocks, pair settings) and reading through it.