hugpy-wrapper (fitevict)

hugpy's per-box wrapper: an evict-to-fit front door over llama.cpp (and transformers). Point any OpenAI client's base URL at it and one address becomes a self-managing model server.

A /v1 call carries only a model name — never a location. For each request the wrapper resolves the name to its weights, runs an evict-to-fit plan against what is resident and what is being called, places the model (launch / evict), forwards the call, splits the model's <think> out of the answer, and records the call — raw stamps and engine geometry — in its own store, which it also reads back to place the next model better.

It needs nothing but an install, a host and a port: no database server, no other hugpy package, no sudo. The decision core is pure stdlib; hardware facts come from nvidia-smi; model facts from the files on disk.

Install

pip only delivers the package; fitevict install is the install.

pip install hugpy-wrapper            # or: pipx install hugpy-wrapper

fitevict install --user --port auto                  # your own service, no root
sudo "$(command -v fitevict)" install --system --port auto   # a system service (root once)
fitevict install ... --dry-run                       # print the plan, change nothing

No port is ever assumed: give a number, or auto for a free one (it is printed and written to the config). The host defaults to 127.0.0.1; --host 0.0.0.0 serves the network.

--system sets up the hugpy layout — each hugpy component runs as its own user, and the group hugpy shares only the model store:

--user uses the environment you installed into, ~/.config/hugpy/wrapper.env and a user unit (systemctl --user; loginctl enable-linger keeps it running after logout).

Every step is safe to repeat: a re-run repairs or upgrades and keeps the box key and the port.

fitevict status [--system|--user]                    # installed? answering? store size?
fitevict uninstall [--system|--user] [--purge]       # --purge also removes config, store, user

Other options: --worker NAME (default: the host name), --engine SPEC (pip-install an engine), --package SPEC (the code to install, e.g. a wheel file), --env KEY=VALUE (repeatable).

Platforms: Linux with systemd today (Ubuntu, Debian, Kali, Fedora, Arch …). macOS (launchd) and Windows services are next; the command says so rather than half-installing.

The engine

GGUF models need llama-server. It is looked up as LLAMA_SERVER_BIN, then under HUGPY_ENGINE_DIR, then on PATH — and the service's PATH starts with its own environment, so an engine installed there (fitevict install … --engine <spec>) is found. fitevict install reports whether one resolves. Without it the wrapper still runs; GGUF calls fail until an engine is there.

pip install "hugpy-wrapper[transformers]"   # the transformers engine (into its own venv)

Use

curl -s http://127.0.0.1:<port>/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"<name>","messages":[{"role":"user","content":"hi"}],"max_tokens":128}'

Any OpenAI client works: base_url=http://<host>:<port>/v1. The model is named, not located; the wrapper finds it, fits it, serves it, logs it.

API keys

Keys look like hpk_<id>_<secret> and are sent as Authorization: Bearer … — what every OpenAI client sends for its api_key, so a user just sets OPENAI_API_KEY=hpk_….

• The box key (hpk_box_…, admin) is written once by fitevict install into the service config. It is the root of trust for the box; hugpy-core uses it to manage the box.

• More keys, made through the running service:

  fitevict keys create --name laptop [--scope use|admin] [--expires-days 90]   # printed once
  fitevict keys list [--all]
  fitevict keys revoke <id>
  

  Only a hash of each key is stored. Revoked and expired keys stop working at once.

Who needs a key:

What counts as inside is FITEVICT_TRUSTED_NETS (default: loopback, 10/8, 172.16/12, 192.168/16, fd00::/8, fe80::/10). Behind a proxy (nginx, a tunnel), every request arrives from 127.0.0.1: a request with forwarding headers counts as outside unless its immediate peer is listed in FITEVICT_TRUSTED_PROXIES — only then is X-Forwarded-For believed. A public proxy can never make the internet "local".

The store (telemetry)

The wrapper keeps every call and every measurement in one SQLite file it creates on first use (FITEVICT_HOME, else systemd's state directory, else the per-OS data directory; 0700 / 0600). Its placement decisions read it back: the measured CUDA context, llama.cpp's projections, serving growth, KV structure, host memory, a model's largest real use. Every measurement points at the call that took it; views compute the figures. A damaged file is set aside and recreated — everything in it is re-learnable. Schema changes ship as numbered migrations.

hugpy-core (or anything with an admin key) works with a box through the management API:

GET  /v1/telemetry?table=call_log&since=<id>&limit=<n>   rows after a cursor (call_log, resource_log,
                                                         evictions, operator_actions) + next / more
GET  /v1/telemetry/state                                 GPU / host state, residents, models, facts, settings
POST /v1/operator/actions   {"action","model_key"}       evict | evict-all | free-ram | remove-model | restart
GET  /v1/operator/actions/<id>
GET|POST /v1/settings/pair  {"model_key","settings","env_profile"}   a model's knobs on this box
POST /v1/settings/quant-facts {"model_key","file","size_bytes","kv_cost"}   size + KV ticks
POST /v1/settings/env-profile {"name","base","packages"}
GET|POST /v1/keys, POST /v1/keys/<id>/revoke

A call's own knobs (model@options, request fields) still override the stored settings per call. python -m fitevict.observability.calls_view --port <n|auto> serves a read-only page of the latest calls on 127.0.0.1 (it shows every call's bodies: run it as the service's user, keep it local).

Every model kind, one front door

Chat (GGUF via llama-server, transformers via tf_child), vision (vision_child) and every non-chat engine go through the SAME placement (evict-to-fit), the SAME call log row and the SAME residents / models catalog. Non-chat engines are runners on one generic child (lifted/task_child.py); each is one file in lifted/runners/ plus one RunnerSpec:

Binary outputs are written to FITEVICT_ARTIFACT_DIR and returned in the endpoint's OpenAI shape; the call log stores their exact bytes + sha256, never the bytes. Uploads are spooled to FITEVICT_SPOOL_DIR and removed after the call.

Env profiles (lifted/profiles.py): a runner's extra pip deps live in a per-profile venv at <engine>/envs/<name> — an overlay on the box's worker base (FITEVICT_TF_PYTHON) or, when a model pins its own torch (chatterbox), an isolated venv. Declared in runners.PROFILES, overridable with POST /v1/settings/env-profile; a model is pinned to one with POST /v1/settings/pair (env_profile). A profile is built on first use, off the request path; until it is ready the call fails as data (503 profile_materializing). GET /profiles reports every profile's state.

Cores

Every engine gets its own cores, claimed once at load from the plan's RAM:VRAM split (CORES.md):

• An engine with nothing in RAM gets one core, which is all it uses.
• An engine reading weights from RAM gets the cores that still make it faster, by the box's measured bandwidth curve. Until that curve is known, it gets every free core.
• Two engines never share a core.
• Cores busy with other work are avoided, and one core stays reserved for the wrapper and the OS.

llama.cpp gets -t N --cpu-mask … --cpu-strict 1. The engine process is confined to the claim by CPU affinity, which needs no privilege.

Each call row records the claim (cpu_set, threads, cpu_source) and the CPU time the call used (cpu_user_ms, cpu_sys_ms). GET /v1/cores shows the box's topology, per-CPU load, every engine's claim and the fitted speed constants.

Per model you can raise the claim with cpu_priority or pin it with cpu_cores (a count or a CPU list) in POST /v1/settings/pair. A pin is honoured even when it has to take busy cores, and the row records what it took.

Configuration

fitevict install writes these into the service config; you rarely set them by hand.

Layout

fitevict/
  core/           pure allocator: types, evict (victim selector), plan (staged fit + quant ladder),
                  flex (ctx-band compress, layers-that-fit), host (EngineHost + drive loop)
  estimate/       sizing, predict, ctx_probe
  serve/          front_door (the OpenAI server), manage (management API), auth (API keys),
                  run_engine (juggle harness), variants, gpu_state, worker_storage_ops
  observability/  store (SQLite) + schema/ migrations, resource_log, calls_view, blobs
  cli/            installer (`fitevict`), console (`fitevict-console`)
  adapters/       llama_cpp (measure / price / launch / evict), transformers, vision, tasks,
                  db (call log + measurements over the store)
  lifted/         helpers lifted from hugpy: gguf_inspect, gguf_need, hardware, spill_reserve,
                  model_resolve, native_resolve, procutil, timings, no_think, app_dirs, runners/ …
  imports/        shared constants / functions / classes (no heavy imports)

See MIGRATION.md (package layout), MIGRATION-imports.md (shared imports layer) and MIGRATION-store.md (store, management API, installer, API keys).

Test

pip install "hugpy-wrapper[test]"
pytest          # every test runs on its own empty store; nothing touches a real one