hugpy-wrapper
The hugpy wrapper: an OpenAI /v1 front door, the queue, placement and eviction, and the engines (llama.cpp and the kg_engines children). It reads hugpy-discovery only through its socket and export.
Install: the deb (ops.sh full-install); hugpy-wrapper system install|status|uninstall.
Settings: /etc/hugpy/wrapper/settings.json. State: /var/lib/hugpy/wrapper, /var/lib/hugpy/resources.
Source: assembled from /srv/hugpy/app/bundles/KNOWN_GOOD/TEST.
The sort, and what core is sent
Spec: /srv/hugpy/app/bundles/KNOWN_GOOD/DOCS. Each of these predicts or reports; the run still decides every step at the row's turn.
- Output-length range (
queue/rows.pyoutput_range): a call's engine time is taken over the fewest to the most tokens the unit's own finished calls emitted, capped bymax_tokens. A unit with fewer thanOUT_HISTORY_MINsuch calls keeps[1, max_tokens]. - The plan and its capacity rows (
queue/plan.py,GET /v1/queue/plan): the waiting rows in the order the run will pop them, each priced against what the rows ahead leave behind. A row is added wherever a capacity is reached: the card (evict-to-fit, with who goes and the seconds) or the worker drive (reap-to-fit, with the bytes short). The pop is the plan's first call row. A row's place in the plan is saved at dispatch (p_ready_s,p_eta_s,p_position,p_capacity). - Stages (
call_stages): one row per finished stage of a call, named as inSTAGE_ORDER(the operator's order as data). - GPU-only engines share one logical CPU (
engines/llama.pychoose_cpus); an engine with host bytes keeps whole cores to itself. - Telemetry for core (
queue/telemetry.py):GET /v1/telemetry/state,GET /v1/telemetry,GET /v1/telemetry/resources, andPOST /v1/requests/{id}/cancel./healthcarriesservice,workerandversion. The wrapper reports itself ashugpy-wrapper-nativewith a routable catalog, not an inventory of copies. Call bodies and replies are not sent.workerin the settings file names the box when it is not the hostname.
Tests: python -m pytest tests (they run the staged source and open no live state). Live checks: /srv/hugpy/app/tools/live-checks/hugpy-wrapper/spec-finish-20261110/.
The fit: where a unit goes (0.1.185.5)
One spill, on one scale, for every unit of every engine (resources/fit.py).
- The tiers are ordered by speed. At start-up the wrapper works out what a token pays per GiB read from the
card, the RAM, the swap and the drive (the time prediction's own rates) and fills the fastest first. Table
tiers. - A unit can be spilled or it cannot. A GGUF is walked down the tiers a whole tensor at a time, the tensors every token reads first. Every other unit sits whole on the fastest tier that holds it.
- The slowest tier is the floor: the model's own file. It holds anything, so every unit has a place.
- Feasibility is stored at start-up (and when the units change): every unit's bytes against the box, as the
range each tier can hold of it.
GET /v1/feasibility,hugpy-feasibility. Tablefeasibility. - A call asks the same fit with what is free now; evict-to-fit asks it with the card as if the residents
were gone, and gives up residents in order of
L = p * delta. A unit the card can never hold evicts nobody. - A load that runs out of memory is recorded (table
fit_failures) and the call goes on at the next place. - Names exist at the input and the output only. A call may send
"placement":gpu-only,gpu-max,ram-only,ram-max,swap-only,swap-max,disk_mmap-only,disk_mmap-max,moeorauto. It becomes a bound on the fit; a bound that cannot be met is dropped and the call is answered where it can be. What is shown (the dashboard, the feasibility output) is named from the bytes.
The queue is rows per engine (0.1.185.6)
A call waits for its own resident and for nothing else. Getting a unit placed and loaded (reap, evict-to-fit,
load) is one row at a time, because two calls deciding at once would take the same free bytes; once a unit is
loaded its engine answers on its own, and the next row that can run is taken at once. A row can run unless the
resident it is for is answering another call. A resident that is answering a call is nobody's victim: evict-to-fit
gives up idle residents only, and a busy one is not re-homed or restarted. The plan (GET /v1/queue/plan) carries
one clock for placing and loading and one per resident.
The 4-bit representation is one more place (0.1.185.7)
A unit that cannot be spilled can still be loaded smaller: in 4-bit, where its engine can do that. The fit prices it the way it prices the stored weights and takes it when it is the faster place by the tiers' speeds.
- Who can: the engine registry says which engines load 4-bit and which parts of a unit they quantize
(
transformers: the whole model;videogen: the transformer, text encoder and unet), and only when the engine's own environment importsbitsandbytes(asked once at start;engines.bnb4,engines.bnb4_note). - How big: discovery's
api_unit_representations_v1(hugpy-discovery 0.1.7.12): each part's bytes as stored, as bf16 and as 4-bit. A part the engine quantizes that discovery has no 4-bit figure for is priced at the ratio of the unit's own estimated parts. Stored at start-up infeasibility.bnb4_bytes, with its basis. - Where: whole on the card; 4-bit runs nowhere else. The engine is started with
--bnb4. - The caller's word:
"bnb4": falsenever,"bnb4": truewhenever it fits, absent = by speed. - A 4-bit load that runs out of memory is recorded apart from the stored ones, and the call goes on stored.
Per-model options: an explicit allocation (0.1.185.9)
What can be chosen for a model is exactly what the fit takes (queue/options.py), and a choice is an explicit
allocation: per tier, the most of the model it may hold (the sliders), and its representation.
GET /v1/models/options?model=ID: the sliders (each tier's ends on this worker, where a call would put the model now, where it was set), the names as slider positions for this model here (presets), the 4-bit representation (available or why not), the model's quants, and what is chosen.POST /v1/models/options {"model": ID, "placement": NAME | "limit_gib": {tier: GiB} | "bnb4": bool}: keep a choice ("clear": trueremoves it). A name sets the sliders to where it sits;limit_gibmoves any of them. Also"ctx"(tokens, up to what the model was trained for) and"quant"(which quant the model's name means)."model": "*"is the worker's default, under every model's own choice.- A call may send the same three fields; its own words come over what is kept for the model.
What is kept (table selections) is the sliders, in bytes. A name (gpu-only, gpu-max, ram-only ...) is a
definition of slider positions, said at the input and made again at the output from the bytes.
Speed or quality ("goal", docs/wrapper-auto.md). By speed alone the smaller representation always wins. Under
speed (the goal when none is said) the fit takes 4-bit when it is the faster place, and a model's bare name means
its fastest quant, each priced by what one pass pays where the fit puts it on this worker. quality is the user's
limit on that: the weights as stored and the quant as chosen. It can be said for one model or for every model
("model": "*"); a chosen quant, or a call that names its quant, is never changed.
Context is a range; the engines' environments are the wrapper's own (0.1.185.10)
- Context.
ctxis the most context wanted andctx_minthe least that will do. The unit is fitted at the low end, which says how deep down the tiers it has to reach at all; it is then loaded with the largest context in the range that reaches no deeper. A context chosen high is never why a load fails or spills. A call that needs more than the range raises it. - The model list (
GET /v1/models) carries, per model, what is chosen for it (options: allocation, context, 4-bit, goal, quant, and how it is loaded now), for a client's columns. - Engine environments live where the wrapper is installed (
/opt/hugpy/wrapper/engines, the defaultprofile_home), never on a models drive: a models drive is meant to be full. The installer creates the directory and says so whenprofile_homesits on the same drive as the worker or central models. - The wrapper prepares them itself. An engine that can load 4-bit has a profile whose manifest carries what that
needs (
bitsandbytes,accelerate); at start the wrapper brings those profiles to their manifests: packages are added to a working environment in place (it is never cleared for this), a profile never built is built, one an engine is running on waits, and nothing is attempted with less than 2 GiB free. Until an environment is ready its engine is not offered 4-bit, andengines.bnb4_notesays why.
Auto is never an unknown (0.1.185.12)
- Context on Auto is the most that fits, up to all the model was trained for: it grows into the room beside weights that sit whole on a tier and never takes their place; weights that are spilled even at the least context get the default one. Room is made by eviction for the default context only; a longer one gives no resident up.
- MoE is a choice for a model with routed experts (
"moe": true | false): on keeps its experts off the card. - What the wrapper does with each model is in the model list (
options.placed: where its bytes go against the box, as a name and as a share per tier), so a client never shows a bare "Auto"; the explicit sliders stand at it. - The engine's overhead on the card is measured against the context a load really had, not a default one.
A new host in one command: system install --host --central --worker --engines
hugpy-wrapper system install takes what a new worker has to be told (system/engines.py):
--host auto|ADDR: the address the front door binds and advertises;autois this host's LAN address. Without it a fresh install listens on 127.0.0.1 only.--central DIRand--worker DIR: where this host's model files are kept, and the fast local directory they are staged on to be loaded (central_root,worker_root); both are entered as discovery's drives (hugpy-discovery system install --central DIR --worker DIR). The wrapper places nothing until both are declared.--engines llama:llama_serveris thehugpy-llama-serverpackage's binary.--engines all: llama, and what every other engine runs on, fetched underprofile_home: a python 3.11 base interpreter with the packages pinned insystem/engine_base.json(base/), the ollama release (ollama/) and ComfyUI's source (comfyui/app, itsvenvthe comfy profile). About 15 GiB; the service then builds each engine's own environment at start (about 55 GiB when all are built). An interpreter or a ComfyUI the settings already name, and that is there, is kept.
Each option sets the settings it names and nothing else; a step already done is skipped, so the command can be run again. A whole worker from nothing:
curl -fsSL https://hugpy.ai/apt/setup.sh | sudo bash -s -- --install wrapper \
&& sudo apt-get install -y hugpy-discovery \
&& sudo hugpy-wrapper system install --host auto --central /srv/hugpy/models --worker /srv/hugpy/worker --engines all
Metadata
Release files for hugpy-wrapper 0.1.195
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| hugpy_wrapper-0.1.195.tar.gz | 481.1 kB | Without provenance | Details for hugpy_wrapper-0.1.195.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| hugpy_wrapper-0.1.195-py3-none-any.whl | Python 3 | none | any | Details for hugpy_wrapper-0.1.195-py3-none-any.whl |
Total release size: 1.0 MB
Release files / hugpy_wrapper-0.1.195.tar.gz
| Download URL | hugpy_wrapper-0.1.195.tar.gz |
|---|---|
| Size | 481.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
058a84757c04b5d66cf9ba1d4ba54825cc61b852a09a38324e9272ed2faf99bc
|
|
BLAKE2b-256 checksum How to use checksums |
d5501f671ce889b777e5588e17b035bb133d73487ba8e1a4f94823a89559e878
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.3
|
Release files / hugpy_wrapper-0.1.195-py3-none-any.whl
| Download URL | hugpy_wrapper-0.1.195-py3-none-any.whl |
|---|---|
| Size | 550.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b6b56f7026c0477efd1e8bd96349962cc61fe1ad791cfb34621a57b509f0a65c
|
|
BLAKE2b-256 checksum How to use checksums |
5e5e36cdb5cfed7fa305793cd099c62c73e90822a79adfb2134d01575896c706
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.3
|
Security for hugpy-wrapper 0.1.195
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.