hugpy-wrapper (fitevict)

hugpy's per-box wrapper: an evict-to-fit front door over llama.cpp (and transformers). Point any OpenAI client's base URL at it and one address becomes a self-managing model server.

A /v1 call carries only a model name — never a location. For each request the wrapper resolves the name to its weights, runs an evict-to-fit plan against what is resident and what is being called, places the model (launch / evict), forwards the call, splits the model's <think> out of the answer, and records the call — raw stamps and engine geometry — in its own store, which it also reads back to place the next model better.

It needs nothing but an install, a host and a port: no database server, no other hugpy package, no sudo. The decision core is pure stdlib; hardware facts come from nvidia-smi; model facts from the files on disk.

Install

pip only delivers the package; fitevict install is the install.

pip install hugpy-wrapper            # or: pipx install hugpy-wrapper

fitevict install --user --port auto                  # your own service, no root
sudo "$(command -v fitevict)" install --system --port auto   # a system service (root once)
fitevict install ... --dry-run                       # print the plan, change nothing

No port is ever assumed: give a number, or auto for a free one (it is printed and written to the config). The host defaults to 127.0.0.1; --host 0.0.0.0 serves the network.

--system sets up the hugpy layout — each hugpy component runs as its own user, and the group hugpy shares only the model store:

--user uses the environment you installed into, ~/.config/hugpy/wrapper.env and a user unit (systemctl --user; loginctl enable-linger keeps it running after logout).

Every step is safe to repeat: a re-run repairs or upgrades and keeps the box key and the port.

fitevict status [--system|--user]                    # installed? answering? store size?
fitevict uninstall [--system|--user] [--purge]       # --purge also removes config, store, user

Other options: --worker NAME (default: the host name), --engine SPEC (pip-install an engine), --package SPEC (the code to install, e.g. a wheel file), --env KEY=VALUE (repeatable).

Platforms: Linux with systemd today (Ubuntu, Debian, Kali, Fedora, Arch …). macOS (launchd) and Windows services are next; the command says so rather than half-installing.

The engine

GGUF models need llama-server. It is looked up as LLAMA_SERVER_BIN, then under HUGPY_ENGINE_DIR, then on PATH — and the service's PATH starts with its own environment, so an engine installed there (fitevict install … --engine <spec>) is found. fitevict install reports whether one resolves. Without it the wrapper still runs; GGUF calls fail until an engine is there.

pip install "hugpy-wrapper[transformers]"   # the transformers engine (into its own venv)

Use

curl -s http://127.0.0.1:<port>/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"<name>","messages":[{"role":"user","content":"hi"}],"max_tokens":128}'

Any OpenAI client works: base_url=http://<host>:<port>/v1. The model is named, not located; the wrapper finds it, fits it, serves it, logs it.

API keys

Keys look like hpk_<id>_<secret> and are sent as Authorization: Bearer … — what every OpenAI client sends for its api_key, so a user just sets OPENAI_API_KEY=hpk_….

• The box key (hpk_box_…, admin) is written once by fitevict install into the service config. It is the root of trust for the box; hugpy-core uses it to manage the box.

• More keys, made through the running service:

  fitevict keys create --name laptop [--scope use|admin] [--expires-days 90]   # printed once
  fitevict keys list [--all]
  fitevict keys revoke <id>
  

  Only a hash of each key is stored. Revoked and expired keys stop working at once.

Who needs a key:

What counts as inside is FITEVICT_TRUSTED_NETS (default: loopback, 10/8, 172.16/12, 192.168/16, fd00::/8, fe80::/10). Behind a proxy (nginx, a tunnel), every request arrives from 127.0.0.1: a request with forwarding headers counts as outside unless its immediate peer is listed in FITEVICT_TRUSTED_PROXIES — only then is X-Forwarded-For believed. A public proxy can never make the internet "local".

The store (telemetry)

The wrapper keeps every call and every measurement in one SQLite file it creates on first use (FITEVICT_HOME, else systemd's state directory, else the per-OS data directory; 0700 / 0600). Its placement decisions read it back: the measured CUDA context, llama.cpp's projections, serving growth, KV structure, host memory, a model's largest real use. Every measurement points at the call that took it; views compute the figures. A damaged file is set aside and recreated — everything in it is re-learnable. Schema changes ship as numbered migrations.

hugpy-core (or anything with an admin key) works with a box through the management API:

GET  /v1/telemetry?table=call_log&since=<id>&limit=<n>   rows after a cursor (call_log, resource_log,
                                                         evictions, operator_actions) + next / more
GET  /v1/telemetry/state                                 GPU / host state, residents, models, facts, settings
POST /v1/operator/actions   {"action","model_key"}       evict | evict-all | free-ram | remove-model | restart
GET  /v1/operator/actions/<id>
GET|POST /v1/settings/pair  {"model_key","settings","env_profile"}   a model's knobs on this box
POST /v1/settings/quant-facts {"model_key","file","size_bytes","kv_cost"}   size + KV ticks
POST /v1/settings/env-profile {"name","base","packages"}
GET|POST /v1/keys, POST /v1/keys/<id>/revoke

A call's own knobs (model@options, request fields) still override the stored settings per call. python -m fitevict.observability.calls_view --port <n|auto> serves a read-only page of the latest calls on 127.0.0.1 (it shows every call's bodies: run it as the service's user, keep it local).

Every model kind, one front door

Chat (GGUF via llama-server, transformers via tf_child), vision (vision_child) and every non-chat engine go through the SAME placement (evict-to-fit), the SAME call log row and the SAME residents / models catalog. Non-chat engines are runners on one generic child (lifted/task_child.py); each is one file in lifted/runners/ plus one RunnerSpec:

Binary outputs are written to FITEVICT_ARTIFACT_DIR and returned in the endpoint's OpenAI shape; the call log stores their exact bytes + sha256, never the bytes. Uploads are spooled to FITEVICT_SPOOL_DIR and removed after the call.

Env profiles (lifted/profiles.py): a runner's extra pip deps live in a per-profile venv at <engine>/envs/<name> — an overlay on the box's worker base (FITEVICT_TF_PYTHON) or, when a model pins its own torch (chatterbox), an isolated venv. Declared in runners.PROFILES, overridable with POST /v1/settings/env-profile; a model is pinned to one with POST /v1/settings/pair (env_profile). A profile is built on first use, off the request path; until it is ready the call fails as data (503 profile_materializing). GET /profiles reports every profile's state.

Cores

Every engine gets its own cores, claimed once at load from the plan's RAM:VRAM split (CORES.md):

• An engine with nothing in RAM gets one core, which is all it uses.
• An engine reading weights from RAM gets the cores that still make it faster, by the box's measured bandwidth curve. Until that curve is known, it gets every free core.
• Two engines never share a core.
• Cores busy with other work are avoided, and one core stays reserved for the wrapper and the OS.

llama.cpp gets -t N --cpu-mask … --cpu-strict 1. The engine process is confined to the claim by CPU affinity, which needs no privilege.

Each call row records the claim (cpu_set, threads, cpu_source) and the CPU time the call used (cpu_user_ms, cpu_sys_ms). GET /v1/cores shows the box's topology, per-CPU load, every engine's claim and the fitted speed constants.

Per model you can raise the claim with cpu_priority or pin it with cpu_cores (a count or a CPU list) in POST /v1/settings/pair. A pin is honoured even when it has to take busy cores, and the row records what it took.

Configuration

fitevict install writes these into the service config; you rarely set them by hand.

Set any of these at install with fitevict install --env HUGPY_QUANT_FALLBACK=Q6_K,Q4_K_M (repeatable); they are rendered into the service's env file like the rest.

Auto policy

When a call arrives by model name alone (no pin), the front door places it Auto. The fallback rung order, per candidate artifact, is:

gpu-only → MoE CPU-expert split (MoE models) → 4-bit → ram-only → max-ram → max-gpu

then it steps the quant down to the next approved variant of the same model and restarts at gpu-only. "4-bit" is the approved 4-bit representation of the same model: for transformers the bnb 4-bit runtime quant of the artifact; for GGUF an approved Q4-class variant file in the catalog (skipped when there is none). When every rung is exhausted the result is terminal with a specific reason — resource (steps the quant), bad artifact or incompatible engine (both terminal, never a quant change). An identical failed plan (artifact, mode, ctx, kv type) is not retried in the same request without evidence the room changed. The stepping runs live in the placement driver (not just reported): each variant is loaded with its own facts, and the call row records the artifact that actually loaded. The box env HUGPY_QUANT_FALLBACK is the ceiling over which variants may be stepped to (auto | off | an ordered label list); the per-pair quality_alternatives can only narrow or reorder inside it. The applied policy (env value, pair knob, ordered candidates) is written to the decision / plan_note.

Concurrent calls enter placement through an admission queue (HUGPY_QUEUE_POLICY = smart default | fifo). Smart: while the FIFO head waits on an eviction or room, calls that can run now may go ahead — resident-served first (grouped), then fits-without-eviction by shortest time-to-output — but a group is never reordered and a head is bypassed at most HUGPY_QUEUE_BYPASS_MAX times and never past its max_wait_s (aging). GET /v1/queue shows the live queue in order with the reason the next one runs; each call row records its queue position, bypass count and why it ran.

A rung whose weights live in host RAM (MoE split, ram-only, max-ram, max-gpu) is admissible only if its CPU claim would get at least cpu_floor_cores free physical cores (the CPU cliff) — memory fit alone never admits it; a hyperthread sibling is not an extra core.

Among the admissible candidates the goal profile ranks them (minimized left to right): quality → (q, T+R, T); fast → (T, q, R); resident → (q, E_hot, T+R), where q = index in the approved-representation list (0 = selected), T = estimated seconds to complete from the box's measured rates (unmeasured → ladder order, never an invented bandwidth), R = reload cost of displaced residents, E_hot = pinned/high-priority residents displaced. Waiting is a candidate only when a resource change is expected within max_wait_s.

Context has three explicit states per (model × mode × KV type): range (shrink on over [feasible_min, feasible_max]), point (a single feasible tick, shrink off), and infeasible (no tick fits the worker budget — refused, never loaded as a point). A resident loaded below its ctx_max grows back by an idle rebuild when there is demand, sustained headroom with a margin, a cooldown has passed, and it is idle (ctx_restore).

Per-pair knobs (user_settings, sent to the box by core via POST /v1/settings/pair; an absent knob = Auto):

Glossary

One set of shared names for quantities the code already carries. The DB columns and API fields do not change — this maps the existing names to the shared ones.

Layout

fitevict/
  core/           pure allocator: types, evict (victim selector), plan (staged fit + quant ladder),
                  flex (ctx-band compress, layers-that-fit), host (EngineHost + drive loop)
  estimate/       sizing, predict, ctx_probe
  serve/          front_door (the OpenAI server), manage (management API), auth (API keys),
                  run_engine (juggle harness), variants, gpu_state, worker_storage_ops
  observability/  store (SQLite) + schema/ migrations, resource_log, calls_view, blobs
  cli/            installer (`fitevict`), console (`fitevict-console`)
  adapters/       llama_cpp (measure / price / launch / evict), transformers, vision, tasks,
                  db (call log + measurements over the store)
  lifted/         helpers lifted from hugpy: gguf_inspect, gguf_need, hardware, spill_reserve,
                  model_resolve, native_resolve, procutil, timings, no_think, app_dirs, runners/ …
  imports/        shared constants / functions / classes (no heavy imports)

See MIGRATION.md (package layout), MIGRATION-imports.md (shared imports layer) and MIGRATION-store.md (store, management API, installer, API keys).

Test

pip install "hugpy-wrapper[test]"
pytest          # every test runs on its own empty store; nothing touches a real one