Skip to main content

hugpy-wrapper

The hugpy wrapper: an OpenAI /v1 front door, the queue, placement and eviction, and the engines (llama.cpp and the kg_engines children). It reads hugpy-discovery only through its socket and export.

Install: the deb (ops.sh full-install); hugpy-wrapper system install|status|uninstall. Settings: /etc/hugpy/wrapper/settings.json. State: /var/lib/hugpy/wrapper, /var/lib/hugpy/resources. Source: assembled from /srv/hugpy/app/bundles/KNOWN_GOOD/TEST.

The sort, and what core is sent

Spec: /srv/hugpy/app/bundles/KNOWN_GOOD/DOCS. Each of these predicts or reports; the run still decides every step at the row's turn.

  • Output-length range (queue/rows.py output_range): a call's engine time is taken over the fewest to the most tokens the unit's own finished calls emitted, capped by max_tokens. A unit with fewer than OUT_HISTORY_MIN such calls keeps [1, max_tokens].
  • The plan and its capacity rows (queue/plan.py, GET /v1/queue/plan): the waiting rows in the order the run will pop them, each priced against what the rows ahead leave behind. A row is added wherever a capacity is reached: the card (evict-to-fit, with who goes and the seconds) or the worker drive (reap-to-fit, with the bytes short). The pop is the plan's first call row. A row's place in the plan is saved at dispatch (p_ready_s, p_eta_s, p_position, p_capacity).
  • Stages (call_stages): one row per finished stage of a call, named as in STAGE_ORDER (the operator's order as data).
  • GPU-only engines share one logical CPU (engines/llama.py choose_cpus); an engine with host bytes keeps whole cores to itself.
  • Telemetry for core (queue/telemetry.py): GET /v1/telemetry/state, GET /v1/telemetry, GET /v1/telemetry/resources, and POST /v1/requests/{id}/cancel. /health carries service, worker and version. The wrapper reports itself as hugpy-wrapper-native with a routable catalog, not an inventory of copies. Call bodies and replies are not sent. worker in the settings file names the box when it is not the hostname.

Tests: python -m pytest tests (they run the staged source and open no live state). Live checks: /srv/hugpy/app/tools/live-checks/hugpy-wrapper/spec-finish-20261110/.

The fit: where a unit goes (0.1.185.5)

One spill, on one scale, for every unit of every engine (resources/fit.py).

  • The tiers are ordered by speed. At start-up the wrapper works out what a token pays per GiB read from the card, the RAM, the swap and the drive (the time prediction's own rates) and fills the fastest first. Table tiers.
  • A unit can be spilled or it cannot. A GGUF is walked down the tiers a whole tensor at a time, the tensors every token reads first. Every other unit sits whole on the fastest tier that holds it.
  • The slowest tier is the floor: the model's own file. It holds anything, so every unit has a place.
  • Feasibility is stored at start-up (and when the units change): every unit's bytes against the box, as the range each tier can hold of it. GET /v1/feasibility, hugpy-feasibility. Table feasibility.
  • A call asks the same fit with what is free now; evict-to-fit asks it with the card as if the residents were gone, and gives up residents in order of L = p * delta. A unit the card can never hold evicts nobody.
  • A load that runs out of memory is recorded (table fit_failures) and the call goes on at the next place.
  • Names exist at the input and the output only. A call may send "placement": gpu-only, gpu-max, ram-only, ram-max, swap-only, swap-max, disk_mmap-only, disk_mmap-max, moe or auto. It becomes a bound on the fit; a bound that cannot be met is dropped and the call is answered where it can be. What is shown (the dashboard, the feasibility output) is named from the bytes.

The queue is rows per engine (0.1.185.6)

A call waits for its own resident and for nothing else. Getting a unit placed and loaded (reap, evict-to-fit, load) is one row at a time, because two calls deciding at once would take the same free bytes; once a unit is loaded its engine answers on its own, and the next row that can run is taken at once. A row can run unless the resident it is for is answering another call. A resident that is answering a call is nobody's victim: evict-to-fit gives up idle residents only, and a busy one is not re-homed or restarted. The plan (GET /v1/queue/plan) carries one clock for placing and loading and one per resident.

The 4-bit representation is one more place (0.1.185.7)

A unit that cannot be spilled can still be loaded smaller: in 4-bit, where its engine can do that. The fit prices it the way it prices the stored weights and takes it when it is the faster place by the tiers' speeds.

  • Who can: the engine registry says which engines load 4-bit and which parts of a unit they quantize (transformers: the whole model; videogen: the transformer, text encoder and unet), and only when the engine's own environment imports bitsandbytes (asked once at start; engines.bnb4, engines.bnb4_note).
  • How big: discovery's api_unit_representations_v1 (hugpy-discovery 0.1.7.12): each part's bytes as stored, as bf16 and as 4-bit. A part the engine quantizes that discovery has no 4-bit figure for is priced at the ratio of the unit's own estimated parts. Stored at start-up in feasibility.bnb4_bytes, with its basis.
  • Where: whole on the card; 4-bit runs nowhere else. The engine is started with --bnb4.
  • The caller's word: "bnb4": false never, "bnb4": true whenever it fits, absent = by speed.
  • A 4-bit load that runs out of memory is recorded apart from the stored ones, and the call goes on stored.

Per-model options: an explicit allocation (0.1.185.9)

What can be chosen for a model is exactly what the fit takes (queue/options.py), and a choice is an explicit allocation: per tier, the most of the model it may hold (the sliders), and its representation.

  • GET /v1/models/options?model=ID: the sliders (each tier's ends on this worker, where a call would put the model now, where it was set), the names as slider positions for this model here (presets), the 4-bit representation (available or why not), the model's quants, and what is chosen.
  • POST /v1/models/options {"model": ID, "placement": NAME | "limit_gib": {tier: GiB} | "bnb4": bool}: keep a choice ("clear": true removes it). A name sets the sliders to where it sits; limit_gib moves any of them. Also "ctx" (tokens, up to what the model was trained for) and "quant" (which quant the model's name means). "model": "*" is the worker's default, under every model's own choice.
  • A call may send the same three fields; its own words come over what is kept for the model.

What is kept (table selections) is the sliders, in bytes. A name (gpu-only, gpu-max, ram-only ...) is a definition of slider positions, said at the input and made again at the output from the bytes.

Speed or quality ("goal", docs/wrapper-auto.md). By speed alone the smaller representation always wins. Under speed (the goal when none is said) the fit takes 4-bit when it is the faster place, and a model's bare name means its fastest quant, each priced by what one pass pays where the fit puts it on this worker. quality is the user's limit on that: the weights as stored and the quant as chosen. It can be said for one model or for every model ("model": "*"); a chosen quant, or a call that names its quant, is never changed.

Context is a range; the engines' environments are the wrapper's own (0.1.185.10)

  • Context. ctx is the most context wanted and ctx_min the least that will do. The unit is fitted at the low end, which says how deep down the tiers it has to reach at all; it is then loaded with the largest context in the range that reaches no deeper. A context chosen high is never why a load fails or spills. A call that needs more than the range raises it.
  • The model list (GET /v1/models) carries, per model, what is chosen for it (options: allocation, context, 4-bit, goal, quant, and how it is loaded now), for a client's columns.
  • Engine environments live where the wrapper is installed (/opt/hugpy/wrapper/engines, the default profile_home), never on a models drive: a models drive is meant to be full. The installer creates the directory and says so when profile_home sits on the same drive as the worker or central models.
  • The wrapper prepares them itself. An engine that can load 4-bit has a profile whose manifest carries what that needs (bitsandbytes, accelerate); at start the wrapper brings those profiles to their manifests: packages are added to a working environment in place (it is never cleared for this), a profile never built is built, one an engine is running on waits, and nothing is attempted with less than 2 GiB free. Until an environment is ready its engine is not offered 4-bit, and engines.bnb4_note says why.

Auto is never an unknown (0.1.185.12)

  • Context on Auto is the most that fits, up to all the model was trained for: it grows into the room beside weights that sit whole on a tier and never takes their place; weights that are spilled even at the least context get the default one. Room is made by eviction for the default context only; a longer one gives no resident up.
  • MoE is a choice for a model with routed experts ("moe": true | false): on keeps its experts off the card.
  • What the wrapper does with each model is in the model list (options.placed: where its bytes go against the box, as a name and as a share per tier), so a client never shows a bare "Auto"; the explicit sliders stand at it.
  • The engine's overhead on the card is measured against the context a load really had, not a default one.

A new host in one command: system install --host --central --worker --engines

hugpy-wrapper system install takes what a new worker has to be told (system/engines.py):

  • --host auto|ADDR: the address the front door binds and advertises; auto is this host's LAN address. Without it a fresh install listens on 127.0.0.1 only.
  • --central DIR and --worker DIR: where this host's model files are kept, and the fast local directory they are staged on to be loaded (central_root, worker_root); both are entered as discovery's drives (hugpy-discovery system install --central DIR --worker DIR). The wrapper places nothing until both are declared.
  • --engines llama: llama_server is the hugpy-llama-server package's binary.
  • --engines all: llama, and what every other engine runs on, fetched under profile_home: a python 3.11 base interpreter with the packages pinned in system/engine_base.json (base/), the ollama release (ollama/) and ComfyUI's source (comfyui/app, its venv the comfy profile). About 15 GiB; the service then builds each engine's own environment at start (about 55 GiB when all are built). An interpreter or a ComfyUI the settings already name, and that is there, is kept.

Each option sets the settings it names and nothing else; a step already done is skipped, so the command can be run again. A whole worker from nothing:

curl -fsSL https://hugpy.ai/apt/setup.sh | sudo bash -s -- --install wrapper \
  && sudo apt-get install -y hugpy-discovery \
  && sudo hugpy-wrapper system install --host auto --central /srv/hugpy/models --worker /srv/hugpy/worker --engines all

Metadata

Release files for hugpy-wrapper 0.1.209

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for hugpy-wrapper 0.1.209
File Size Uploaded Provenance Actions
hugpy_wrapper-0.1.209.tar.gz 511.1 kB Without provenance Details for hugpy_wrapper-0.1.209.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for hugpy-wrapper 0.1.209
File Interpreter ABI Platform Actions
hugpy_wrapper-0.1.209-py3-none-any.whl Python 3 none any Details for hugpy_wrapper-0.1.209-py3-none-any.whl

Total release size: 1.1 MB

Release files / hugpy_wrapper-0.1.209.tar.gz

Download URL hugpy_wrapper-0.1.209.tar.gz
Size 511.1 kB
Tags Source
SHA-256 checksum
How to use checksums
05c5b6414997cf3f78318fe65b8b8504357b15dd6686315e0154aa858b4b3c9b
BLAKE2b-256 checksum
How to use checksums
add00c58f707c7108b9bd19f2cbdfe0aebc7f15a4f83e5c08bdf50af746a490f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Release files / hugpy_wrapper-0.1.209-py3-none-any.whl

Download URL hugpy_wrapper-0.1.209-py3-none-any.whl
Size 574.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
d57a9f2d49a640d4963ca4db3e835e27d11b2a299f1edf10c28b90408f0b7b95
BLAKE2b-256 checksum
How to use checksums
6b43d90dd24438b56bb9bdc0698413bd392b16bea67b1ce46b4abf28ea43552d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Security for hugpy-wrapper 0.1.209

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.209 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page