hugpy-wrapper (fitevict)

Serving a local PEFT adapter

Give a trained variant its own model directory and model name, with the base model configuration, tokenizer and weight files (these may be symlinks). Set "adapter_dir": "/absolute/path/to/adapter" in that directory's hugpy.json. Relative paths are resolved against the model directory. The adapter must contain adapter_config.json and adapter_model.safetensors; a missing adapter fails launch instead of silently serving the base. The transformers engine's peft dependency loads it on CPU or GPU, including a 4-bit base. Memory pricing includes the unquantized adapter at PEFT's default fp32 promotion size. Keep the adapter immutable under its registered model name so cached sizing stays valid.

hugpy's per-box wrapper: an evict-to-fit front door over llama.cpp (and transformers). Point any OpenAI client's base URL at it and one address becomes a self-managing model server.

A /v1 call carries only a model name — never a location. For each request the wrapper resolves the name to its weights, runs an evict-to-fit plan against what is resident and what is being called, places the model (launch / evict), forwards the call, splits the model's <think> out of the answer, and records the call — raw stamps and engine geometry — in its own store, which it also reads back to place the next model better.

It needs nothing but an install, a host and a port: no database server, no other hugpy package, no sudo. The decision core is pure stdlib; hardware facts come from nvidia-smi; model facts from the files on disk.

A stop is bounded (0.1.102)

SIGTERM unloads the residents under the placement lock; with the lock held by a placement (or a long refresh) the stop sat until systemd's 90 s kill (ae, 2026-10-14, stopping the worker to swap its drive). The lock is now taken with a 15 s bound and the residents are unloaded regardless.

The GGUF engine is a package (0.1.101)

hugpy-wrapper now Recommends hugpy-llama-server: llama.cpp's llama-server (CUDA) under /opt/hugpy-llama, which the engine discovery finds (/opt/*/bin/llama-server). Before, a fresh install served GGUF only if a legacy build happened to sit under /srv/*/.local/share/hugpy/engine (ae, 2026-10-14: cleaning that directory made every GGUF call fail with FileNotFoundError).

A pull never duplicates what the worker holds (0.1.100)

pull-model is refused when this worker already holds the bytes anywhere (its own weight_hashes, any key): a second copy of the same model re-keys it (discovery suffixes duplicates) and the fleet loses the model's identity — seen on ae pulling Echo-Mini it already held on another drive.

A re-run keeps the host; pulls is a cursor table (0.1.99)

fitevict install --system without --host keeps the configured FITEVICT_HOST the way it keeps the port and the token — the apt upgrade's postinst had put a LAN-bound worker (joined by hugpy-core workers add) back on loopback, where its central could not reach it. pulls joins the cursor tables central mirrors (/v1/telemetry?table=pulls&since=).

A worker pulls a model on central's order (0.1.98)

Operator action pull-model {model_key, params: {source_url, headers, file}} — central names the source (its own verified copy on a central drive, served at its /v1/transfer, or another worker holding the files) and this worker pulls with the stdlib puller (serve/worker_pull.py): Range reads per chunk, each chunk's sha256 against the source's chunksums as it lands, the whole file's sha256 against the source's, a .part staging file resumed where its chunks verify, an atomic rename. The copy lands in this worker's own model root as <MODELS_HOME>/<owner>/<repo>/ with the marker beside the weights, then refresh-models discovers it and the hasher proves it. Refused when the files are already here or there is no room within the storage tier ("reap first"). Every file is a pulls row (migration 0013: bytes, seconds, ok, error — the measured rate central reads).

Derivation rules are part of the marker signature (0.1.96, 0.1.97)

A sidecar marker is re-derived when the model's config signature changes — and now when the derivation rules change (DERIVE_RULES in the signature), so 0.1.95's cache-snapshot identity reaches the markers 0.1.93 derived. 0.1.97: the re-derivation happens ON READ of a sidecar whose signature is stale (before, only ensure_marker re-derived, and a dir that already had a sidecar never reached it). A hugpy.json beside the model is never touched.

A huggingface cache snapshot is a model too (0.1.95)

A model that lives in ~/.cache/huggingface/hub/models--owner--repo/snapshots/<rev> is symlinks into the repo's blobs/. The byte server's listing skipped every symlink, so its manifest was empty and central's copier could never advance past it; the derived marker named it snapshots/<rev>. Now the models--owner--repo dir is part of the path jail (jail_roots), the snapshot's files are served through their blobs, and the marker derives owner/repo from the cache dir's name (hub_identity). Hidden files (.cache/, a downloader's .lock, .gitattributes) are never offered: they are not the model.

kv_seqs is a number (0.1.94)

llama.cpp's newer load log says 1/1 seqs; the K2 reader stored that text in kv_structure.kv_seqs (declared INTEGER — SQLite keeps what it is handed) and central's typed mirror then refused the row and the whole sync with it. The reader and the writer now bind the first number; migration 0012 repairs what 0.1.93 stored.

The box serves its own files; a remote central; storage over the drives (0.1.93)

Phase 4 D2 (operator 2026-10-14): central and other workers pull a model's files FROM THE WORKER with abstract_transfer — GET /v1/transfer/<model_key>/manifest, …/file?path= (HTTP Range → 206, 1 MiB stream), …/chunksums?path= (32 MiB sha256 chunks, sidecar-cached), a path jail inside the model dir, one concurrency cap per box (HUGPY_TRANSFER_MAX, 3) — lifted from the orbit's hugpy_storage.transfer_api, gated by the box key. Join 3: fitevict install --central URL --central-key KEY [--advertise HOST] registers the worker with a REMOTE central itself (POST /api/fleet/workers); a loopback-bound wrapper refuses, said so. The storage tier (0.1.92) now sums every drive our models live on, not only the shared store's.

Capacity tiers (0.1.92)

Operator 2026-10-14: device ≥ env designation ≥ central's setting, for VRAM, RAM, cores and storage, in the resource's smallest unit, with unmanaged encroachment charged to the device. lifted/capacity.py; install flags --vram-max/--ram-max/--cpu-max/--storage-max (refused above the device); GET /limits carries tiers; POST /limits accepts cpu_limit and storage_limit and REFUSES a setting above its parent tier with the reason. The VRAM designation caps the card; the cores cap cuts the claim's free list. See CORES.md and /srv/hugpy/directions/auto-multiplexing/response/"3 - capacity tiers (2026-10-14, proposal).md".

The fit map honours the cores (0.1.90)

/harness/v1/models (the fit map central's allocation reads) drops the host-resident placements of every quant when the box's cores say no — the same rule as the front door's gate and the verdict cells — so central never allocates a RAM placement the box would refuse. See CORES.md.

The compressed profile is declared by the package (0.1.89)

A fresh install could not load GLM-4.7-Flash-NVFP4: the env profile that gives the child compressed-tensors had been made by hand in the store and the purge took it. The profile is now DECLARED in lifted/runners.PROFILES and a compressed-tensors checkpoint maps to it by default (adapters/llama_cpp.default_profile_for); the first load kicks its build (503 profile_materializing, a fact), /profiles reports it ready (~20 s), the next load runs from it. The pair knob env_profile still overrides.

Hashes once; pairs known elsewhere (0.1.87) · evict-to-fit over cores (0.1.88)

0.1.87: every weight file is hashed ONCE (weight_hashes, the hasher thread woken by the model map; GET /v1/hashes) — content identity for the fleet model DB's exact-duplicate rule; and POST /v1/facts/import lets central hand this box the facts another worker measured for a (model, quant file) it does not hold, so GET /v1/verdicts prices those pairs too (on_disk: false, origin_worker). 0.1.88: an idle engine's cores are obtainable by the next claim (donor re-pinned or evicted), the constant CPU floor is gone (the knob, else S3, else no basis), the gate never fails open and refuses a single-thread start beside busy engines, and every host-resident verdict cell carries its cores (wanted / obtainable / predicted tok/s). See CORES.md.

The wrapper defines the worker; central finds it (0.1.85, 0.1.86)

Operator 2026-10-14: "the wrapper install should be what defines a worker. and that worker should either be discoverable by an existing central, or a post installed central." /health now carries the worker's identity — {"status": "ok", "service": "hugpy-wrapper", "worker": <FITEVICT_WORKER>, "version": <installed>} — and central registers a wrapper under the name it reports (0.1.85). The install's tenth step, central, registers this worker with a central already on the box: when /etc/hugpy/core.endpoint.json exists it calls hugpy-core register --url --key --name; a central installed later reads wrapper.endpoint.json itself; a central elsewhere runs hugpy-core register. A central that refuses never fails the wrapper install — the worker is usable on its own (0.1.86). A worker keeps its name across reinstalls; its port may change, and central accepts the move because the wrapper at the new address says who it is.

Discovery evaluates every marker; a load no longer blocks the box (0.1.84)

Operator 2026-10-14: "is discover models just taking whatever? how is it not evaluating the hugpy.json for correctness?" It was: the file beside the model was central's stamp and trusted as long as it parsed; only JSON damage was recovered and one lie (a text-generation stamp the config contradicts) overridden. A 2026-07-27 stub with tasks: null passed. Now marker.evaluate_marker compares every marker with the on-disk evidence (classify: framework, tasks, primary task, MoE) and the engine's abilities (transformers.bnb_available: can this engine run bitsandbytes at all) and with the marker schema (marker_schema, now 2). A marker that disagrees — or predates the schema — is RE-DERIVED from disk, the hub-origin facts the disk cannot produce carried over (hub_id, hub, manifest, admission, …), the stale stamp recorded under superseded with the reasons, and written beside the model (sidecar when read-only); the catalog row says hugpy_json_source: rederived and why. Evaluated once per (dir, config, marker mtime). A 4-bit placement is offered only when the engine has bitsandbytes (DAN-Qwen3-1.7B had been offered one and failed on ImportError).

And the placement flaw: a six-minute load held the serialized placement section, so three calls that fit now waited behind it. front_door._yield_placement hands the section on while a launched child loads (lock and queue released for the wait, re-taken for the load's bookkeeping); the loading model is pinned busy (never an eviction victim) and its launch budget stays reserved out of the room (inflight_reserved_bytes counts loading children of every engine). The host brackets its health wait with _load_wait (sets child.loading).

0.1.82 shipped a LAN relax of the management routes; 0.1.83 reverts it

The management routes (/v1/telemetry*, /v1/queue, /v1/settings/*, /v1/operator/*, /v1/keys, /v1/cores, /v1/verdicts) are machine-to-machine: loopback, or an admin key (core and the router hold the box key). 0.1.82 let the trusted networks manage without a key; it contradicted the designed rule and its tests and was reverted the same night. An operator acts through the console's login, never through this door.

Nothing is written that did not change; readings are silent, events are loud (0.1.80–0.1.81)

Operator 2026-10-14: "please don't heartbeat, this is long since deprecated." The store's change counter is what every long-poll reader wakes on (GET /v1/telemetry/state?wait=N&since=K: the router's box link, hugpy-core's collector). The residents publisher used to rewrite every seat on each tick with a fresh observed_at, so the counter moved every second while serving and every ten seconds idle — a heartbeat in all but name, and every reader re-synced on it. db._replace_rows now diffs against the stored rows: an identical set is not a transaction, an unchanged row is not rewritten, and the stamp column means "last changed". A quiet box moves the counter only when something actually changed — and (0.1.81) only when that something is an EVENT. The GPU / host / CPU watch (every 250 ms, NVML and /proc) and a serving resident's live bytes are SENSOR READINGS: they are stored (store.transaction(notify=False)) but move no counter, so a card's bytes drifting, the host's page cache moving or a core's busy fraction changing wakes nobody. Events are what wake readers: a load, an eviction, a call row, a fact, a resident's state or seat changing, an operator action, or a process appearing on / leaving the card (the watch promotes that one reading to an event). Measured before: 81 counter moves in 20 s on an idle box; after: none.

The ask is compared, never the rung (0.1.79)

0.1.78 compared the resident's request with the NEXT call's contract as it stood inside the auto ladder — already carrying that attempt's rung (alloc_mode, moe, n_cpu_moe) and the window derivation (ctx_start, ctx_ladder, ctx_need) — so two identical Auto calls still read as "knobs changed" and a 60 GB resident was reloaded on every call. _request_view now strips everything a placement derives (keeping a pinned / preset / retry window, which is part of the ask) and _place_locked compares the new call's own snapshot of its ask with the resident's.

A refused window is a measured fact; a busy resident is never torn down (0.1.78)

Two things the GLM first-call run showed. (1) The probe took three points between 3k and 7k tokens and a 54k fill that OOMed the card; the line through the three points then said 141k fits. A line beyond its last point is a projection, not a measurement: sizing.window_refused now keeps every window at or above a failed probe point of a variant out of reach (Auto, the pinned planner, ctx_fits_budget) until a SUCCESSFUL reading at or above that window exists — for every variant (gpu / gpu:bnb4 / moe), not only GGUF gpu-only. (2) A foreground prompt arrived while a grading call was being generated; the Auto ladder had written its outcome (alloc_mode = ram-only) into the resident's contract, so the new Auto call read "knobs changed the load contract" and evicted the resident mid-generation (the grading call answered 502). Now the compare is against the REQUEST the resident answered (contract_requested, taken before the auto ladder runs), so Auto's own outcome never reads as a knob change; and a resident that truly must be reloaded under another contract is never stopped while a call is on it (_wait_idle_for_replacement: wait within the ticket's max_wait_s, then fail the new call as data with 503 and leave the resident serving).

Fused experts are materialized from the packed checkpoint (0.1.77)

Measured 2026-10-14 on GLM-4.7-Flash-NVFP4 (transformers 5.3, compressed-tensors 0.19): the first generation under the expert split answered "15 com Comments com …". The load log said why — transformers fuses each layer's per-expert projections into one experts.gate_up_proj / experts.down_proj tensor, but has no converter for the per-expert NVFP4 weight_packed / weight_scale / weight_global_scale tensors: it reported them UNEXPECTED and the fused tensors of layers 2–45 MISSING (newly initialized). Forty-four of forty-seven MoE layers served random experts. The child now runs three materialization steps after from_pretrained, before any placement (lifted/dequant.py): materialize_compressed (eager decompress of the matched modules + offload wrappers removed, 0.1.76), materialize_packed_experts (every fused expert tensor whose layer has packed per-expert weights in the checkpoint is filled, expert by expert, with compressed-tensors' own unpack + dequantize — bit-identical to its NVFP4PackedCompressor.decompress — in the fused [gate; up] order), and dequantize_float8 (the FP8 fused experts, 0.1.75). A tensor whose checkpoint shape does not match the fused one is left as loaded and reported. The resident is then the plain bf16 model the load history prices (GLM: 59.9 GB counted — 4.4 GB on the card, 55.6 GB of experts in RAM).

No ratios left on the sizing path (0.1.76)

Operator 2026-10-13: "make sure absurd constants are not being applied." An audit of the live sizing path found one residual ratio: the pinned / partial planner budgeted host RAM at 95 % of the measured free figure (Policy.ram_safety_frac = 0.95), the rung the transformers expert split takes. It is now 1.0 — RAM is budgeted at the measured MemAvailable with swap as the second tier (sizing.host_fit), exactly like VRAM. Weight and KV terms were already exact or measured (GGUF margins 1.0, cushion 0, VRAM / RAM reserves default 0, VRAM safety 1.0); the "x1.15 header formula" survives only as a label in the full core's estimate fallback, which the basic core does not load. And one more "measurement wins" rule, the transformers twin of the GGUF K2 rewrite: once the child has served two or more calls whose cache tally grew with the tokens held, the observed bytes per token (kv_bytes_engine / slot_tokens, median of the last 20) replace the config-geometry ladder (basis: transformers child cache (observed, N calls)); a preallocated cache that reads the same bytes at every token count is not taken as a per-token fact.

The grader writes the facts from reality (0.1.75)

Operator ruling 2026-10-13 (evening): "On the very first successful (or failed-but-informative) load of a model, create a persistent load history record for that model + placement class. No prior synthetic 'facts' size is trusted after that point." The packed on-disk size of a compressed-tensors dir (NVFP4 / FP8 + scales) is not what loads — transformers materializes it as bf16 (GLM-4.7-Flash-NVFP4: 21.26 GB packed, 57.85 GB resident) — and the planner believed the packed number.

• The child counts its weights. tf_child / vision_child tally the resident's parameters and buffers per device after loading (decompression, dequantization, 4-bit packing, adapters, the expert split included) and serve them on GET /props as weights {gpu_bytes, cpu_bytes, total_bytes, params, dtypes}.
• The loading call row carries them (doctrine 012: the row holds the raw measurement, the fact table points): weights_bytes is now the counted total, with the split in weights_gpu_bytes / weights_host_bytes (new columns, migration 0010), the process VRAM after the load and the host RSS beside them. A failed load points too (its row says which rung refused) but never displaces a measured load.
• The history is keyed by placement class — tf_measure_facts (worker, key, variant, ctx), variant gpu / gpu:bnb4 / moe / ram — backfilled from the pointed load rows. tf_measure resolves the counted weights, the load and peak bytes, the host RSS and the load's outcome.
• Sizing reads the history first. sizing.tf_need prices a class from its own counted weights (the whole model on the card; the split's card weights + experts in RAM; RAM-only's host weights); a class that never loaded is priced from any class's measured total; only before any measured load does the safetensors header figure stand. The launch budget (adapters.transformers.footprint, the expert split's --max-memory) and the core claim read the same history.
• The packed hint learns. tf_facts.load_bytes sizes a compressed-tensors dir at its bf16 materialization (weight_packed ×4, FP8 weight ×2, scales 0) so the very first Auto decision already steps past the card; a tf_structure row written before 0.1.75 is refreshed once.
• The grader is the measurement engine. A grading or smoke-test call still drives the first load and the post-serve probes, but the probe now sweeps the DEFAULT window ± HUGPY_PROBE_SPAN ticks (default 1) and the requested window (ctx_probe.targets(ladder, default, requested)), keys a transformers expert-split resident as its own moe class, and — the exemption — once the resident's history is complete (ctx_probe.complete: every point has a row, the full-context fill measured) grading and smoke-test calls skip the probes. Real traffic keeps refreshing the call log.
• Leftover FP8 is dequantized in the child (lifted/dequant.py): a compressed-tensors checkpoint whose fused expert tensors no scheme matched keeps float8 weights no Ampere card or CPU matmul takes (grouped_mm … got Float8_e4m3fn); after from_pretrained every float8 parameter is dequantized with its weight_scale (per expert, read back from the shards when the module no longer carries it). A parameter whose scale cannot be found is left float8 and reported.

GLM-4.7-Flash-NVFP4 stays in the catalog (operator: "a model that is called is a model that is called; only cull if it is broken or cannot fit any resource allocation") — Auto sizes it from the history and takes the expert split.

Time-to-output is the queue's goal (0.1.74)

Operator ruling 2026-10-13: "time-to-output should be the goal over fifo; a base time to process-init should be predetermined for any model in the queue — the historical time to loaded once eviction/occupancy allows; historical time to output for all occupants processing; if unknown then fifo for that initial call; a maximum window after which the queued item is auto next in line." serve/eta.py reads those figures off the call rows the box already keeps (nothing new is measured): load_s = median weights_loaded_at - weights_staged_at, prompt_ms / decode_ms = median per-token timings, out_tokens = median emitted tokens (capped by the call's max_tokens). The front door tracks in-flight calls (_INFLIGHT: started + predicted seconds) and the queue's estimator turns them into each ticket's ready_in_s. admission.choose_next policy eta orders by eta_of = ready-in + load + output, keeps unknown models in their arrival slots, keeps a group's order, puts background tickets last, and promotes a ticket whose window (max_wait_s, default 120 s) has passed or would be passed by its predicted start. Default policy is now eta; HUGPY_QUEUE_POLICY=smart|fifo restores the earlier rules.

Background tickets for grading and probes (0.1.73)

Grading, smoke tests, benchmarks and warm-ups ride the same admission queue as real calls but as BACKGROUND tickets (metadata.intent in grading / smoke-test / benchmark / warmup, or metadata.priority = background): every foreground ticket precedes them, a background ticket never bypasses a head waiting on room and never ages ahead of a real call. This is what lets hugpy-core (0.2.13) grade a model from the call log — it asks the rubric items as background calls and reads the verdicts back off call_log — while the box keeps answering operators first.

Serving leaves the placement lock (0.1.72)

Until now the forward/stream ran under the placement lock, so a box served ONE call at a time across every model: a short answer from a resident model blocked another resident model and blocked every load. Now the lock covers only the evict/load decision. A serving model is pinned busy first (the eviction planner already never selects a mid-generation resident; the polite-shrink path already skips it), then the section is handed to the next admission ticket. Same-model calls are serialized on a per-model serve slot sized to the engine (1 for a transformers child, -np for llama-server), so the wait is the front door's and the call's timings stay honest. The queue classifies a busy resident as resident_now (its wait is the slot, not room).

The room a placement sees now subtracts, for each live transformers resident, its budgeted GPU need minus the bytes it holds at that moment (inflight_reserved), because its KV cache grows per token and is released after the generation; without this a placement made between two of its calls could take the room its next call needs. GGUF engines preallocate KV and compute at load and reserve nothing. The reservation shows in GET /limits (ceiling.inflight_reserved) and each resident's gpu_budget in GET /residents.

The environment log rides the model's own rows (0.1.71)

Nothing new is written anywhere: every env-profile event is an operator_actions row with action = 'env-profile' and the model_key it is attributed to — the cursor table hugpy-core already mirrors as w_operator_actions — so inference can pinpoint which pip landed in which venv for which model. result is {event, profile, base, packages, lock, dir, python, overlay_of, error, previous}; state is done (or failed for a failed build); requested_by is the caller or profile:<name>.

Events: declared (POST /v1/settings/env-profile; carries model_key when the body names one), assigned / unassigned (POST /v1/settings/pair with an env_profile that differs from the pair's current one; previous = the old name), build-requested (a model's load gated on a profile not yet built — attributed to that model), building, built (the frozen pip lock, the venv dir and python, overlay_of = the worker base it overlays) and build-failed (the pip error). Query the model's environment history with

SELECT requested_at, state, json_extract(result, '$.event') AS event, result
  FROM operator_actions WHERE action = 'env-profile' AND model_key = ?
 ORDER BY id

Migration 0009 widens the action CHECK and adds an index on (model_key, id); existing operator history is kept. Any operator_actions consumer that filters by action keeps working — the new action never enters the requested state, so the operator-actions runner never sees it.

The transformers expert split lands (0.1.68 – 0.1.70)

A transformers MoE footprint now carries priced allocation candidates, so the pinned / partial planner (core/preferred.py) accepts the expert split: the whole model on the card, RAM-only, and — with the pair's MoE knob on — the split (n_gpu_layers -1, n_cpu_moe = the routed expert count, non-expert tensors + KV + overhead on the card, routed experts in RAM). With alloc_mode explicit and moe on and nothing else pinned, the split IS the explicit choice (no percentage required); the Auto rung pins it the same way. Measured on ae with Qwen3.6-35B-A3B (256 experts): 5.95 GB predicted / 6.11 GB measured on the card, the experts (66 GB) in RAM, 4.1–4.5 tok/s.

The split's context band is priced like every other mode: sizing._mode_need and ctx_fits_budget price the transformers split with tf_need(…, "moe") (before 0.1.70 they used the GGUF layer-split pricing, so the envelope reported "facts missing" and the split loaded at the 2048 history default). Auto now lands it at the feasible maximum of the ladder (262,144 on the box above), and /v1/verdicts reports the split's ctx_max per KV type by walking the ladder.

A refused Auto rung no longer leaves the call row as a finished failure while the next rung loads: the row returns to pending before the next attempt, so Core's replay cursor keeps re-pulling it until the call really ends.

Facts for every model and the blanket preferences (0.1.65, 0.1.66)

Every model is priced: a transformers dir gets its KV ladder from its config geometry (decoder-only, encoder-decoder and nested decoder configs; diffusers and config-less dirs are no-context heads with KV = 0), so sizing.decide answers for it the same way as for a GGUF. A transformers MoE has the expert split as a placement: the routed experts stay in host RAM and execute on the CPU (tf_child --moe-split), everything else on the card; it is priced from the per-tensor split (tf_facts.expert_split) and offered by Auto right after GPU-only, exactly as the GGUF --n-cpu-moe rung.

Pair knobs from directions/auto-multiplexing: allowed_placements (a subset of gpu-only, moe, 4bit, ram-only, max-ram, max-gpu; Auto never offers a rung outside it), wait_before_quality (wait within max_wait_s for a call in flight to free room before an approved quality change), resident_priority (the eviction band once loaded) and latency_target_s (a measured response estimate above it ranks after every plan within it). They ride the contract and GET /v1/verdicts reports the policy in auto_policy.

The box blanket (operator 2026-10-14). The preferences are a fitevict-level knob, not only a per-model one: GET/POST /v1/settings/box holds the box-wide defaults (auto_goal, tolerance, quality_alternatives, allowed_placements, wait_before_quality, resident_priority, latency_target_s, max_wait_s, cpu_floor_cores and the ctx knobs — pins and per-file choices stay per pair). Every pair inherits them; a pair's own row overrides key by key. GET /v1/settings/pair shows settings (own), box (blanket) and effective (what a call runs under). The blanket is the pair_settings row keyed *.

Telemetry additions (0.1.63, 0.1.64)

GET /v1/verdicts[?model_key=] (0.1.64) reports the pair verdicts hugpy-core and the console read: per quant file and placement mode, the priced figures, the context ceiling and window per KV cache type, the Auto pick and whether the MoE split is offered. A settings write flushes the cache.

GET /storage now lists every drive that holds a discovered model (drives[]: mount, device, filesystem, capacity, the models it contains). GET /v1/telemetry/tables and GET /v1/telemetry/rows?table=… give a read-only page of any table or view of the wrapper store (API key secrets are withheld); hugpy-core's Databases tab reads them through the router relay.

Install

pip only delivers the package; fitevict install is the install.

For a single system install command on Debian or Ubuntu, build the local APT package from this source and install the resulting file:

BUILD_PYTHON=/srv/hugpy/venv/bin/python ./packaging/debian/build.sh
sudo apt install ./build/debian/hugpy-wrapper_0.1.61_all.deb

The .deb bundles the wrapper wheel, creates its own service environment, installs and starts hugpy-wrapper.service, and checks its local health. It does not register the host with Hugpy Core or add a router box. On a fresh install it chooses a free loopback port and writes the endpoint to /etc/hugpy/wrapper.endpoint.json; upgrades reuse the saved port and token. The package does not bundle an inference engine; configure an engine separately before serving models. apt remove hugpy-wrapper stops the service while retaining config, state, and models. apt purge hugpy-wrapper removes wrapper config and state, but retains the shared model store. The package is built locally; these commands do not publish to PyPI.

pip install hugpy-wrapper            # or: pipx install hugpy-wrapper

fitevict install --user --port auto                  # your own service, no root
sudo "$(command -v fitevict)" install --system --port auto   # a system service (root once)
fitevict install ... --dry-run                       # print the plan, change nothing

No port is ever assumed: give a number, or auto for a free one (it is printed and written to the config). The host defaults to 127.0.0.1; --host 0.0.0.0 serves the network.

--system sets up the hugpy layout — each hugpy component runs as its own user, and the group hugpy shares only the model store:

--user uses the environment you installed into, ~/.config/hugpy/wrapper.env and a user unit (systemctl --user; loginctl enable-linger keeps it running after logout).

Every step is safe to repeat: a re-run repairs or upgrades and keeps the box key and the port.

fitevict status [--system|--user]                    # installed? answering? store size?
fitevict uninstall [--system|--user] [--purge]       # --purge also removes config, store, user

Other options: --worker NAME (default: the host name), --engine SPEC (pip-install an engine), --package SPEC (the code to install, e.g. a wheel file), --env KEY=VALUE (repeatable).

Platforms: Linux with systemd today (Ubuntu, Debian, Kali, Fedora, Arch …). macOS (launchd) and Windows services are next; the command says so rather than half-installing.

The engine

GGUF models need llama-server. It is looked up as LLAMA_SERVER_BIN, then under HUGPY_ENGINE_DIR, then on PATH — and the service's PATH starts with its own environment, so an engine installed there (fitevict install … --engine <spec>) is found. fitevict install reports whether one resolves. Without it the wrapper still runs; GGUF calls fail until an engine is there.

pip install "hugpy-wrapper[transformers]"   # the transformers engine (into its own venv)

Use

curl -s http://127.0.0.1:<port>/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"<name>","messages":[{"role":"user","content":"hi"}],"max_tokens":128}'

Any OpenAI client works: base_url=http://<host>:<port>/v1. The model is named, not located; the wrapper finds it, fits it, serves it, logs it.

API keys

Keys look like hpk_<id>_<secret> and are sent as Authorization: Bearer … — what every OpenAI client sends for its api_key, so a user just sets OPENAI_API_KEY=hpk_….

• The box key (hpk_box_…, admin) is written once by fitevict install into the service config. It is the root of trust for the box; hugpy-core uses it to manage the box.

• More keys, made through the running service:

  fitevict keys create --name laptop [--scope use|admin] [--expires-days 90]   # printed once
  fitevict keys list [--all]
  fitevict keys revoke <id>
  

  Only a hash of each key is stored. Revoked and expired keys stop working at once.

Who needs a key:

What counts as inside is FITEVICT_TRUSTED_NETS (default: loopback, 10/8, 172.16/12, 192.168/16, fd00::/8, fe80::/10). Behind a proxy (nginx, a tunnel), every request arrives from 127.0.0.1: a request with forwarding headers counts as outside unless its immediate peer is listed in FITEVICT_TRUSTED_PROXIES — only then is X-Forwarded-For believed. A public proxy can never make the internet "local".

The store (telemetry)

The wrapper keeps every call and every measurement in one SQLite file it creates on first use (FITEVICT_HOME, else systemd's state directory, else the per-OS data directory; 0700 / 0600). Its placement decisions read it back: the measured CUDA context, llama.cpp's projections, serving growth, KV structure, host memory, a model's largest real use. Every measurement points at the call that took it; views compute the figures. A damaged file is set aside and recreated — everything in it is re-learnable. Schema changes ship as numbered migrations.

hugpy-core (or anything with an admin key) works with a box through the management API:

GET  /v1/telemetry?table=call_log&since=<id>&limit=<n>   rows after a cursor (call_log, resource_log,
                                                         evictions, operator_actions) + next / more
GET  /v1/telemetry/state                                 GPU / host state, residents, models, facts, settings
POST /v1/operator/actions   {"action","model_key"}       evict | evict-all | free-ram | remove-model | restart
GET  /v1/operator/actions/<id>
GET|POST /v1/settings/pair  {"model_key","settings","env_profile"}   a model's knobs on this box
POST /v1/settings/quant-facts {"model_key","file","size_bytes","kv_cost"}   size + KV ticks
POST /v1/settings/env-profile {"name","base","packages"}
GET|POST /v1/keys, POST /v1/keys/<id>/revoke

A call's own knobs (model@options, request fields) still override the stored settings per call. python -m fitevict.observability.calls_view --port <n|auto> serves a read-only page of the latest calls on 127.0.0.1 (it shows every call's bodies: run it as the service's user, keep it local).

Every model kind, one front door

Chat (GGUF via llama-server, transformers via tf_child), vision (vision_child) and every non-chat engine go through the SAME placement (evict-to-fit), the SAME call log row and the SAME residents / models catalog. Non-chat engines are runners on one generic child (lifted/task_child.py); each is one file in lifted/runners/ plus one RunnerSpec:

Binary outputs are written to FITEVICT_ARTIFACT_DIR and returned in the endpoint's OpenAI shape; the call log stores their exact bytes + sha256, never the bytes. Uploads are spooled to FITEVICT_SPOOL_DIR and removed after the call.

Env profiles (lifted/profiles.py): a runner's extra pip deps live in a per-profile venv at <engine>/envs/<name> — an overlay on the box's worker base (FITEVICT_TF_PYTHON) or, when a model pins its own torch (chatterbox), an isolated venv. Declared in runners.PROFILES, overridable with POST /v1/settings/env-profile; a model is pinned to one with POST /v1/settings/pair (env_profile). A profile is built on first use, off the request path; until it is ready the call fails as data (503 profile_materializing). GET /profiles reports every profile's state.

Cores

Every engine gets its own cores, claimed once at load from the plan's RAM:VRAM split (CORES.md):

• An engine with nothing in RAM gets one core, which is all it uses.
• An engine reading weights from RAM gets the cores that still make it faster, by the box's measured bandwidth curve. Until that curve is known, it gets every free core.
• Two engines never share a core.
• Cores busy with other work are avoided, and one core stays reserved for the wrapper and the OS.

llama.cpp gets -t N --cpu-mask … --cpu-strict 1. The engine process is confined to the claim by CPU affinity, which needs no privilege.

Each call row records the claim (cpu_set, threads, cpu_source) and the CPU time the call used (cpu_user_ms, cpu_sys_ms). GET /v1/cores shows the box's topology, per-CPU load, every engine's claim and the fitted speed constants.

Per model you can raise the claim with cpu_priority or pin it with cpu_cores (a count or a CPU list) in POST /v1/settings/pair. A pin is honoured even when it has to take busy cores, and the row records what it took.

Configuration

fitevict install writes these into the service config; you rarely set them by hand.

Set any of these at install with fitevict install --env HUGPY_QUANT_FALLBACK=Q6_K,Q4_K_M (repeatable); they are rendered into the service's env file like the rest.

Auto policy

When a call arrives by model name alone (no pin), the front door places it Auto. The fallback rung order, per candidate artifact, is:

gpu-only → MoE CPU-expert split (MoE models) → 4-bit → ram-only → max-ram → max-gpu

then it steps the quant down to the next approved variant of the same model and restarts at gpu-only. "4-bit" is the approved 4-bit representation of the same model: for transformers the bnb 4-bit runtime quant of the artifact; for GGUF an approved Q4-class variant file in the catalog (skipped when there is none). When every rung is exhausted the result is terminal with a specific reason — resource (steps the quant), bad artifact or incompatible engine (both terminal, never a quant change). An identical failed plan (artifact, mode, ctx, kv type) is not retried in the same request without evidence the room changed. The stepping runs live in the placement driver (not just reported): each variant is loaded with its own facts, and the call row records the artifact that actually loaded. The box env HUGPY_QUANT_FALLBACK is the ceiling over which variants may be stepped to (auto | off | an ordered label list); the per-pair quality_alternatives can only narrow or reorder inside it. The applied policy (env value, pair knob, ordered candidates) is written to the decision / plan_note.

Concurrent calls enter placement through an admission queue (HUGPY_QUEUE_POLICY = eta default | smart | fifo). eta (0.1.74): shortest predicted time-to-output first — ready_in (how long until the call's resources free: the same model's in-flight calls, or the soonest occupant when it needs room) + load (the model's measured time-to-loaded, 0 when resident) + output (prompt tokens x its measured ms/prompt-token + expected output tokens x its ms/output-token). Every figure is a median over the model's own last 20 call rows; a model with no decode history yet keeps its arrival slot (fifo for that initial call) and is learned from that row. A group (session) is never reordered; background tickets run after every foreground one; a ticket that has waited its max_wait_s, or whose predicted start would pass it, is next in line (the window). smart and fifo are the 2026-10-08 rules. GET /v1/queue shows the live queue with each ticket's eta and the reason the next one runs; each call row's queue block records position, kind, eta, load, output, ready-in and why it ran.

Background work never delays a real call (0.1.73). A call whose metadata.intent is grading, smoke-test, benchmark or warmup, or whose metadata.priority is background, takes a background ticket: every foreground ticket precedes it, it never bypasses a head waiting on room and never ages ahead of a real call; it runs when nothing foreground waits. The call row's queue block and /v1/queue show background.

Placement is serialized; serving is not (0.1.72). The placement lock covers the evict/load decision only. Once a call is serving, its model is pinned busy (never an eviction victim, never shrunk) and the placement section passes to the next ticket, so calls to different resident models — and another model's load — overlap. Calls to the SAME model queue on its serve slot (1 for a transformers child, FITEVICT_PARALLEL for llama-server); a busy resident still counts as resident_now for the queue. A transformers resident's budgeted GPU need minus what its process holds right now is held out of the room a placement sees (/limits → ceiling.inflight_reserved; /residents → gpu_budget), since its KV grows per token and is freed after the call; GGUF engines preallocate KV + compute and reserve nothing.

A rung whose weights live in host RAM (MoE split, ram-only, max-ram, max-gpu) is admissible only if its CPU claim would get at least cpu_floor_cores free physical cores (the CPU cliff) — memory fit alone never admits it; a hyperthread sibling is not an extra core.

Among the admissible candidates the goal profile ranks them (minimized left to right): quality → (q, T+R, T); fast → (T, q, R); resident → (q, E_hot, T+R), where q = index in the approved-representation list (0 = selected), T = estimated seconds to complete from the box's measured rates (unmeasured → ladder order, never an invented bandwidth), R = reload cost of displaced residents, E_hot = pinned/high-priority residents displaced. Waiting is a candidate only when a resource change is expected within max_wait_s.

Context has three explicit states per (model × mode × KV type): range (shrink on over [feasible_min, feasible_max]), point (a single feasible tick, shrink off), and infeasible (no tick fits the worker budget — refused, never loaded as a point). A resident loaded below its ctx_max grows back by an idle rebuild when there is demand, sustained headroom with a margin, a cooldown has passed, and it is idle (ctx_restore).

Per-pair knobs (user_settings, sent to the box by core via POST /v1/settings/pair; an absent knob = Auto):

Glossary

One set of shared names for quantities the code already carries. The DB columns and API fields do not change — this maps the existing names to the shared ones.

Layout

fitevict/
  core/           pure allocator: types, evict (victim selector), plan (staged fit + quant ladder),
                  flex (ctx-band compress, layers-that-fit), host (EngineHost + drive loop)
  estimate/       sizing, predict, ctx_probe
  serve/          front_door (the OpenAI server), manage (management API), auth (API keys),
                  run_engine (juggle harness), variants, gpu_state, worker_storage_ops
  observability/  store (SQLite) + schema/ migrations, resource_log, calls_view, blobs
  cli/            installer (`fitevict`), console (`fitevict-console`), assist (`fitevict-assist`)
  adapters/       llama_cpp (measure / price / launch / evict), transformers, vision, tasks,
                  db (call log + measurements over the store)
  lifted/         helpers lifted from hugpy: gguf_inspect, gguf_need, hardware, spill_reserve,
                  model_resolve, native_resolve, procutil, timings, no_think, app_dirs, runners/ …
  imports/        shared constants / functions / classes (no heavy imports)

See MIGRATION.md (package layout), MIGRATION-imports.md (shared imports layer) and MIGRATION-store.md (store, management API, installer, API keys).

Small coreless assistant

fitevict-assist searches a drive with a bounded filesystem walk, then uses the wrapper's direct OpenAI endpoint for optional small-model ranking and chat. It does not require hugpy-core. The bundled command can fetch Qwen's official Apache-2.0 Qwen2.5-0.5B-Instruct Q4_K_M GGUF (491 MB) into MODELS_HOME; the download is checked against its published SHA-256 before it is made visible.

fitevict-assist bootstrap
fitevict-assist scan --query 'small coder'       # discover mounted drives automatically
fitevict-assist scan /mnt/models --query 'small coder'
fitevict-assist scan /mnt/models --query 'specific model name' --add
fitevict-assist add /other/drive/my-model
fitevict-assist call 'Which model did we just add?'
fitevict-assist call 'Explain this model' --model My-MoE-GGUF \
    --quant Q4_K_M --alloc max-gpu --moe --ctx high --knob n_cpu_moe=12

Without a root argument, scan checks mounted drives, Hugpy stores, Hugging Face caches, and known local harness locations. It also checks model paths in running llama.cpp, vLLM and KoboldCpp command lines. Each result records the source hints that led to it. These hints are verified against files on disk; they do not make an unknown model runnable by themselves. Scanning has depth, directory-count and result limits. add makes a catalog symlink, establishes hugpy.json from disk evidence (beside the model if writable, otherwise in a wrapper sidecar), and requests an immediate catalog refresh. It does not copy model weights. Unsupported weight-only folders are shown by scan but refused by add. call passes model options and explicit per-call knobs to wrapper, so its existing fit and feasibility checks remain in force. The assistant never executes a path or command returned by the small model.

Test

pip install "hugpy-wrapper[test]"
pytest          # every test runs on its own empty store; nothing touches a real one

Find models outside the configured store

fitevict scout searches mounted drives and known model caches with a bounded directory walk. It reports GGUF, Transformers, and Diffusers model directories, including model paths named by running llama*, Ollama, vLLM, ComfyUI, and Kobold processes. It reads filenames and small config files; it does not load weights or require a language model.

fitevict scout --root /mnt/16T_toshiba/llm_storage
fitevict scout --root /mnt/nvmes/2T_samsung_990/hot990/aeb/models \
  --attach /path/to/owner/model --models-home /var/lib/hugpy/models

--attach names one model directory and may be repeated. It creates a link in the wrapper's owner/repo model layout and attempts to derive a hugpy.json marker from the model's on-disk evidence. Its JSON result reports whether the marker was available. The external weights are left in place. Refresh the running wrapper's catalog after attachment with its refresh-models operator action, or restart the wrapper. The wrapper's direct /v1 endpoint supports model variants such as @moe, @4bit, @max-gpu, and @fit without core.

Automatic local discovery

The wrapper checks conventional Python virtualenv and Conda locations, verifies engine imports as the service account, and reuses a compatible environment. fitevict engines --system reports candidates and the selection. An explicit FITEVICT_TF_PYTHON overrides discovery; automatic selections are cached and rechecked. Engine discovery does not register a Hugpy worker.

The catalog also checks conventional model directories and Hugging Face hub caches, including HF_HOME, HF_HUB_CACHE, HUGGINGFACE_HUB_CACHE, and TRANSFORMERS_CACHE. It follows cached snapshots and checks indexed weight shards. External discoveries reuse files in place without writing model markers. Search is bounded to common locations, five directory levels and 4096 directories; private or unusual paths still require configuration and service read access. Discovery confirms local files and engine imports, not successful inference.

Optional terminal harness

The TUI, chat loop, sessions and agent tools are packaged separately as hugpy-harness. Installing hugpy-wrapper does not install or run them. Install the separate local harness package to use hugpy-harness launch or hugpy-harness chat. It connects over HTTP and also works from another host. Existing workspace chat histories remain compatible.

fitevict setup --harness hugpy-harness --model MODEL_ID --write connection.json generates a connection configuration; launch it with hugpy-harness --config connection.json. Configuration helpers for aider and OpenCode remain available without installing the Hugpy harness.

The wrapper's /v1/telemetry/resources endpoint supplies lightweight resource snapshots to clients. The optional harness displays these alongside chat using /resources and /resources full.