hugpy_core

Basic Core (0.2)

The default hugpy-core runtime is now the basic fleet API and console. It registers existing wrappers, dispatches inference through hugpy-router, shows resources and models, and retains consolidated execution history in its own Postgres database (0.3: a socket-only cluster the install provisions through the bundled toolserver, see BASIC-CORE.md). It does not start a brain, planner, model transfer service, a toolserver node, media application, or a local wrapper.

The console is the original @hugpy/ui application shell (front door, Docs, console tabs, Media/Video/Fleet links) built from its source of record at /srv/hugpy/app/ui/source (npm run build:basic there copies the bundle into hugpy_core/basic/static). Status, Compute, Models, API, Evictions, Metrics, Calls, Databases and Addons read the Core's derived tables (see BASIC-CORE.md); Compute keeps the original VRAM / RAM / CPU / Storage dropdowns and lists each worker's models with their knobs inline. Every other tab, link and feature of the original UI stays visible and is shown explicitly as not installed, naming the addon or arm package that provides it (/api/fleet/addons, /api/arms). The ui/ copy inside this package is an older divergent snapshot kept for reference only.

Build and install the local Debian package on Ubuntu 24.04 / Python 3.12:

BUILD_PYTHON=/srv/hugpy/venv/bin/python ./packaging/debian/build.sh
sudo apt install ./build/debian/hugpy-core_0.2.12_amd64.deb
hugpy-core status

The package bundles its Python dependencies and installs offline into /opt/hugpy-core/venv. The router currently reuses the wrapper's lightweight Python store/auth utilities; this library dependency does not install a wrapper service or inference engine. Only hugpy-core.service is created. An existing local wrapper is registered automatically using its saved endpoint. Its service is neither restarted nor reconfigured.

The installer chooses an unused loopback port on first installation and retains it on upgrades. /etc/hugpy/core.endpoint.json records the console/API URL. /etc/hugpy/core.env contains the Core admin credential; do not print or share that file. State and worker credentials are private to the hugpy-core service account under /var/lib/hugpy-core. Package removal retains configuration and history. The previous PostgreSQL database is not changed or migrated.

Use the console's API page for the client address. For example:

hugpy-harness --base http://127.0.0.1:<core-port>

Harness 0.1.2 understands named fleet resource snapshots and retains local conversations. The OpenAI-compatible base URL is <core-url>/v1. Direct wrapper connections remain supported independently.

See the basic architecture and API for boundaries, deployment, and current limitations. The old full runtime and installer remain explicitly available as hugpy-core legacy ... with the hugpy-core[legacy] dependencies. Legacy features have not yet been converted into independently installable addons; the UI labels them as planned, not available.

Legacy implementation notes (pre-0.2)

The sections below describe the retained legacy runtime and its prior deployment. They do not describe what the default basic installation starts.

hugpy's central interface: console API, auth, planner (verdicts per model x worker), the DB relay, the arms (video, oracle, help) and, today, the call router. Service on ae: 7002_hugpy_api (port 7002, user solcatcher). Its DB is hugpy_core; each box's front door writes hugpy_wrapper.call_log.

Design record: toolserver instructions/hugpy/design/call-flow-router-2026-10-06.

Current one-command installation

pip install hugpy-core installs the matching wrapper, router, toolserver, and bundled console. On the central host, point its transfer catalog at the central drive when installing the services:

hugpy-core install --user --central-models-home /mnt/16T_toshiba/llm_storage/models

The installer retains that setting on later runs. --models separately names the local wrapper's model store on the current box. The full install builds the wrapper's separate transformers engine so GGUF, transformers, vision, and task models on that box are discovered. Use --no-wrapper-tf for a GGUF-only box.

How a call flows (today)

client
  |
  v
CORE  :7002   (interface + router in one process)
  |  1. logs its own call row (hop core)
  |  2. picks a worker: a box where the model is loaded (residents table),
  |     else the best-ranked box for the pair's verdict
  v
WORKER AGENT  (hugpy_worker, one per box)
  |  3. logs its row (hop worker)
  |  4. model not on this box? pull it from central INSIDE the call
  |     -> its own row (hop central-pull) = the cold-load measurement
  v
WRAPPER FRONT DOOR  (fitevict, :7016 on ae; owns the GPU)
  |  5. logs its row (hop wrapper)
  |  6. place: same contract loaded? reuse. else evict-to-fit + load
  |     (llama-server or a task child, localhost-only ports)
  v
ENGINE  (llama-server / task child)
  |  7. serves
  ^
  back up the same path; each hop closes its own row.
  binaries in and out -> content-addressed blob store, rows keep pointers.

A refusal (won't fit, pull failed, load failed) returns up the path as data. Nothing re-routes it to another box today.

Where it is going (operator direction 2026-10-06)

client
  |
  v
ROUTER  (own package/process; behind core's address, so callers see one address)
  |  a. reads the residents view (derived from call_log the moment a row changes)
  |     and the static verdicts. Nothing else.
  |  b. model loaded on a box with matching settings?
  |        -> proxy straight to that box's FRONT DOOR (skip the worker agent)
  |  c. not loaded anywhere?
  |        -> pick a box whose verdict says it can run
  |        -> that box's worker agent pulls if missing (cold load measured)
  |  d. static verdict says impossible on that box?
  |        -> refuse here: one row, reason + verdict pointer, try the next box
  |  e. the box refuses (live room, load failure)?
  |        -> bounce back: try the next box, one row per decision
  v
WRAPPER FRONT DOOR  (thin when the model is loaded: reuse, forward, measure, log)
  v
ENGINE

CORE  (interface: console, auth, planner/verdicts, arms, help) - not on the hot path

Rules behind it:

• One row per decision, written by the hop that made it. Core never forwards a call it knows is impossible just to have it logged further down.
• A model missing from a box is not a refusal: the pull happens inside the call and is the cold-load measurement.
• With no core/router in front, the wrapper is first: it serves or refuses (the reason as data). Bounce-back is the job of whatever router/core is in front.
• A changed pair setting on a loaded model evicts and reloads it when idle.

Benchmark call-log history

POST /llm/benchmark/run opens a core call-log row before validation or planning. Accepted runs keep that row pending until their asynchronous outcome is known. Rejected plans retain their constraint reasons in the core response and error. The response and X-Hugpy-Request-Id expose its request ID.

Benchmark history/status read call_log; the JSON state and run archives are no longer read or written. Existing files are left intact as legacy evidence and are not converted into invented historical call records. Grading entries store call pointers and verdicts. History resolves answers and measurements from the referenced execution records. Each benchmark inference core row points back to the orchestration row, and its execution pointer identifies the wrapper row. Pending benchmark rows from a previous core process are marked interrupted.

The brain (failure auto-evaluation)

Ruling: toolserver instructions/hugpy/brain/brain-established-2026-10-06. A brain lives in core: always present, callable, and it evaluates every model-load failure automatically. hugpy-agent, when installed, adds features and access on top (the agent-station TUI); without it, this brain does the work.

• Event: a hugpy_wrapper.residents row whose load_error changes fires pg_notify('model_rows'); brain.ensure() (started by app.main) listens and evaluates it. No polling.
• Classify: rules first, from the legacy taxonomy (hard_load_failure, vram_fit, engine_unavailable, unreachable, vision_needs_slot, missing_dependency, other), plus pip-shaped signals (pip install X, No module named 'x', transformers' "requires the following packages").
• Infer: the brain model is asked only when the rules leave something open (class other, or an import name with no package name, e.g. cv2 → opencv-python). The call goes through core's own /v1/chat/completions with metadata.intent = "brain-evaluate", so it is a call_log row. The brain never overrules a class the rules determined.
• Record: every evaluation is a Help ticket (help_tickets, kind load-failure), de-duplicated per model/worker/error via occurrences.
• Call it: POST /llm/brain/evaluate {error, worker?, model_key?, file?}; GET /llm/brain shows the model, policy and listener state.

Which model is the brain. The static hugpy-brain model group (see Model groups below) holds the models the brain may run on, in priority order. When the group is empty it fills itself from the grades: every model whose latest verdicts pass the threshold in each listed grader category (default coding 3/3 + logic 3/3 of the hugpy-native-v2 suite's easy/medium/hard tiers), best first. Resolution: the Settings selection (brain.model) → HUGPY_BRAIN_MODEL → the group's first servable member → the best-graded servable model (flagged below threshold). The Help chat uses the same brain (after HUGPY_HELP_MODEL).

Auto-implement gate. Every evaluation carries auto: {eligible, why}. A fix may be implemented without approval only when brain.auto_fix is ticked and it is a pip fix with named packages; a fix read literally off the error needs nothing more, a fix a model inferred also needs that model to meet the threshold. Otherwise it waits for operator approval.

Settings → Policies → Brain (settings_policies):

Not automated yet (same ruling, next step): the venv ladder for missing_dependency — existing env profiles → a new overlay profile → an isolated profile — each candidate test-loaded, then designated (model_workers.env_profile); auto decides whether that happens on its own. Never pip-install into a shared venv.

Model groups (Priority groups panel)

model_groups.py + sql/019_model_groups.sql: the panel's groups are rows in hugpy_core.model_groups (record shape lifted from legacy comms/priority_groups.py: ordered members, ordered workers, enabled), plus static (cannot be deleted or renamed), role and source (operator | auto). GET/POST /llm/model-groups, PUT/PATCH/DELETE /llm/model-groups/<id>. One structural rule, as before: a model key may be in at most one enabled routing group. Role groups (hugpy-brain, role brain) never route, may be empty, and sit outside that rule. Core's request routing does not walk priority groups yet; the list payload says so (routing: false) and resolve answers that honestly. allocate returns 501 (not lifted).

Verdicts priced at the pair's ctx and KV type

planner.refresh_worker passes each pair's user_settings to measured_verdict.price: the verdict is priced at the pair's ctx_pct (k% of the trained window, floored to 1024) and kv_cache_type (f16 / q8_0 / q4_0). A knob write that touches either re-plans that worker at once (modeldb.pair_knobs → planner.replan), so the console's Memory distribution follows the knobs.

The fix ladder (missing dependencies) — fix_ladder.py

Operator design: try every environment on the box that may already fit, else the box's worker base + the new pip install, else keep that new (broken) install attributed and leave the ticket for the operator. Candidates, best first: BUILT profiles whose frozen lock holds every needed package, then DECLARED-but-unbuilt ones whose manifest holds them (the box builds on the test load). Each candidate is attributed (model_workers.env_profile) and test-loaded (envprofiles.run_test, an env-test call, 10 min cap); a failure is reverted, and its log tail (call_log.response_meta.load_log_tail) is read for the next missing package (transformers' "requires the X library" wording included). The new environment is auto-<model> (worker base + packages, up to 3 rounds). Starts on its own when the evaluation is auto-eligible, else on ✓ approve; a core restart resumes it once its worker re-registers. Progress lives on the ticket (detail.ladder). First real fix: Qwen3.5-0.8B on computron → its vision profile (pillow + the box-pinned torchvision).

Allocation determinations

A benchmark lane refused because the worker does not hold the model goes to the brain (brain.constraint_and_file): on another online worker → reroute; in central's catalog → pull (ACTIVATE, automatic when the gate allows, else "✓ pull" on the allocation ticket); nowhere → not available.

Grader tok/s

/llm/model-metrics2 rows carry a throughput block per worker × quant × alloc and /llm/models/<m>/metrics carries worker_averages (Σ tokens / Σ generation seconds over every recorded call), both from the call log. The Grader tab showed "no calls recorded" everywhere because the rows had only a bare tok/s number and worker_averages was a fixed [].

Attach a worker

Core serves a packaged installer at /api/llm/workers/install.sh. In the console, Settings → Boxes issues a single-use enrollment token and an install command. Run that command on the worker. The script installs the wrapper and worker into a user venv, provisions that account's abstract-toolserver node with its own database, enrolls the box, and starts a user service. Set HUGPY_MODELS_HOME before running it to point discovery at a mounted model store. Re-running it keeps the existing node identity and box credential.

The console's /api/llm/events stream sends CPU, RAM, load and GPU changes in small live feeds. The large worker model inventory is sent on connection and when its structure changes.

The base hugpy-core install also includes the media and video arm packages. Each arm owns its endpoint blueprint and its built UI; core discovers the endpoints through the hugpy.arms entry points and loads the arm UI when its page is requested.

Glossary

Shared meanings for the mirrored call-log columns. Core keeps the names hugpy_wrapper gives them; the box measures each value and intake mirrors it into core's copy, so these are read, never recomputed (calllog.py is the one reader).