Skip to main content

hugpy-wrapper

The hugpy wrapper: an OpenAI /v1 front door, the queue, placement and eviction, and the engines (llama.cpp and the kg_engines children). It reads hugpy-discovery only through its socket and export.

Install: the deb (ops.sh full-install); hugpy-wrapper system install|status|uninstall. Settings: /etc/hugpy/wrapper/settings.json. State: /var/lib/hugpy/wrapper, /var/lib/hugpy/resources. Source: assembled from /srv/hugpy/app/bundles/KNOWN_GOOD/TEST.

The sort, and what core is sent

Spec: /srv/hugpy/app/bundles/KNOWN_GOOD/DOCS. Each of these predicts or reports; the run still decides every step at the row's turn.

  • Output-length range (queue/rows.py output_range): a call's engine time is taken over the fewest to the most tokens the unit's own finished calls emitted, capped by max_tokens. A unit with fewer than OUT_HISTORY_MIN such calls keeps [1, max_tokens].
  • The plan and its capacity rows (queue/plan.py, GET /v1/queue/plan): the waiting rows in the order the run will pop them, each priced against what the rows ahead leave behind. A row is added wherever a capacity is reached: the card (evict-to-fit, with who goes and the seconds) or the worker drive (reap-to-fit, with the bytes short). The pop is the plan's first call row. A row's place in the plan is saved at dispatch (p_ready_s, p_eta_s, p_position, p_capacity).
  • Stages (call_stages): one row per finished stage of a call, named as in STAGE_ORDER (the operator's order as data).
  • GPU-only engines share one logical CPU (engines/llama.py choose_cpus); an engine with host bytes keeps whole cores to itself.
  • Telemetry for core (queue/telemetry.py): GET /v1/telemetry/state, GET /v1/telemetry, GET /v1/telemetry/resources, and POST /v1/requests/{id}/cancel. /health carries service, worker and version. The wrapper reports itself as hugpy-wrapper-native with a routable catalog, not an inventory of copies. Call bodies and replies are not sent. worker in the settings file names the box when it is not the hostname.

Tests: python -m pytest tests (they run the staged source and open no live state). Live checks: /srv/hugpy/app/tools/live-checks/hugpy-wrapper/spec-finish-20261110/.

The fit: where a unit goes (0.1.185.5)

One spill, on one scale, for every unit of every engine (resources/fit.py).

  • The tiers are ordered by speed. At start-up the wrapper works out what a token pays per GiB read from the card, the RAM, the swap and the drive (the time prediction's own rates) and fills the fastest first. Table tiers.
  • A unit can be spilled or it cannot. A GGUF is walked down the tiers a whole tensor at a time, the tensors every token reads first. Every other unit sits whole on the fastest tier that holds it.
  • The slowest tier is the floor: the model's own file. It holds anything, so every unit has a place.
  • Feasibility is stored at start-up (and when the units change): every unit's bytes against the box, as the range each tier can hold of it. GET /v1/feasibility, hugpy-feasibility. Table feasibility.
  • A call asks the same fit with what is free now; evict-to-fit asks it with the card as if the residents were gone, and gives up residents in order of L = p * delta. A unit the card can never hold evicts nobody.
  • A load that runs out of memory is recorded (table fit_failures) and the call goes on at the next place.
  • Names exist at the input and the output only. A call may send "placement": gpu-only, gpu-max, ram-only, ram-max, swap-only, swap-max, disk_mmap-only, disk_mmap-max, moe or auto. It becomes a bound on the fit; a bound that cannot be met is dropped and the call is answered where it can be. What is shown (the dashboard, the feasibility output) is named from the bytes.

The queue is rows per engine (0.1.185.6)

A call waits for its own resident and for nothing else. Getting a unit placed and loaded (reap, evict-to-fit, load) is one row at a time, because two calls deciding at once would take the same free bytes; once a unit is loaded its engine answers on its own, and the next row that can run is taken at once. A row can run unless the resident it is for is answering another call. A resident that is answering a call is nobody's victim: evict-to-fit gives up idle residents only, and a busy one is not re-homed or restarted. The plan (GET /v1/queue/plan) carries one clock for placing and loading and one per resident.

The 4-bit representation is one more place (0.1.185.7)

A unit that cannot be spilled can still be loaded smaller: in 4-bit, where its engine can do that. The fit prices it the way it prices the stored weights and takes it when it is the faster place by the tiers' speeds.

  • Who can: the engine registry says which engines load 4-bit and which parts of a unit they quantize (transformers: the whole model; videogen: the transformer, text encoder and unet), and only when the engine's own environment imports bitsandbytes (asked once at start; engines.bnb4, engines.bnb4_note).
  • How big: discovery's api_unit_representations_v1 (hugpy-discovery 0.1.7.12): each part's bytes as stored, as bf16 and as 4-bit. A part the engine quantizes that discovery has no 4-bit figure for is priced at the ratio of the unit's own estimated parts. Stored at start-up in feasibility.bnb4_bytes, with its basis.
  • Where: whole on the card; 4-bit runs nowhere else. The engine is started with --bnb4.
  • The caller's word: "bnb4": false never, "bnb4": true whenever it fits, absent = by speed.
  • A 4-bit load that runs out of memory is recorded apart from the stored ones, and the call goes on stored.

Per-model options: an explicit allocation (0.1.185.9)

What can be chosen for a model is exactly what the fit takes (queue/options.py), and a choice is an explicit allocation: per tier, the most of the model it may hold (the sliders), and its representation.

  • GET /v1/models/options?model=ID: the sliders (each tier's ends on this worker, where a call would put the model now, where it was set), the names as slider positions for this model here (presets), the 4-bit representation (available or why not), the model's quants, and what is chosen.
  • POST /v1/models/options {"model": ID, "placement": NAME | "limit_gib": {tier: GiB} | "bnb4": bool}: keep a choice ("clear": true removes it). A name sets the sliders to where it sits; limit_gib moves any of them. Also "ctx" (tokens, up to what the model was trained for) and "quant" (which quant the model's name means). "model": "*" is the worker's default, under every model's own choice.
  • A call may send the same three fields; its own words come over what is kept for the model.

What is kept (table selections) is the sliders, in bytes. A name (gpu-only, gpu-max, ram-only ...) is a definition of slider positions, said at the input and made again at the output from the bytes.

Speed or quality ("goal", docs/wrapper-auto.md). By speed alone the smaller representation always wins. Under speed (the goal when none is said) the fit takes 4-bit when it is the faster place, and a model's bare name means its fastest quant, each priced by what one pass pays where the fit puts it on this worker. quality is the user's limit on that: the weights as stored and the quant as chosen. It can be said for one model or for every model ("model": "*"); a chosen quant, or a call that names its quant, is never changed.

Context is a range; the engines' environments are the wrapper's own (0.1.185.10)

  • Context. ctx is the most context wanted and ctx_min the least that will do. The unit is fitted at the low end, which says how deep down the tiers it has to reach at all; it is then loaded with the largest context in the range that reaches no deeper. A context chosen high is never why a load fails or spills. A call that needs more than the range raises it.
  • The model list (GET /v1/models) carries, per model, what is chosen for it (options: allocation, context, 4-bit, goal, quant, and how it is loaded now), for a client's columns.
  • Engine environments live where the wrapper is installed (/opt/hugpy/wrapper/engines, the default profile_home), never on a models drive: a models drive is meant to be full. The installer creates the directory and says so when profile_home sits on the same drive as the worker or central models.
  • The wrapper prepares them itself. An engine that can load 4-bit has a profile whose manifest carries what that needs (bitsandbytes, accelerate); at start the wrapper brings those profiles to their manifests: packages are added to a working environment in place (it is never cleared for this), a profile never built is built, one an engine is running on waits, and nothing is attempted with less than 2 GiB free. Until an environment is ready its engine is not offered 4-bit, and engines.bnb4_note says why.

Auto is never an unknown (0.1.185.12)

  • Context on Auto is the most that fits, up to all the model was trained for: it grows into the room beside weights that sit whole on a tier and never takes their place; weights that are spilled even at the least context get the default one. Room is made by eviction for the default context only; a longer one gives no resident up.
  • MoE is a choice for a model with routed experts ("moe": true | false): on keeps its experts off the card.
  • What the wrapper does with each model is in the model list (options.placed: where its bytes go against the box, as a name and as a share per tier), so a client never shows a bare "Auto"; the explicit sliders stand at it.
  • The engine's overhead on the card is measured against the context a load really had, not a default one.

Metadata

Release files for hugpy-wrapper 0.1.188

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for hugpy-wrapper 0.1.188
File Size Uploaded Provenance Actions
hugpy_wrapper-0.1.188.tar.gz 430.3 kB Without provenance Details for hugpy_wrapper-0.1.188.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for hugpy-wrapper 0.1.188
File Interpreter ABI Platform Actions
hugpy_wrapper-0.1.188-py3-none-any.whl Python 3 none any Details for hugpy_wrapper-0.1.188-py3-none-any.whl

Total release size: 942.4 kB

Release files / hugpy_wrapper-0.1.188.tar.gz

Download URL hugpy_wrapper-0.1.188.tar.gz
Size 430.3 kB
Tags Source
SHA-256 checksum
How to use checksums
5226c268fd900d61d8320c69e7c51e0600dc1f4bed6058d3f3d97a5c933e99e3
BLAKE2b-256 checksum
How to use checksums
afb098e7f4ceb2355bb67a881aecd3fb24228a67142318ca5ab76abc47630453
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Release files / hugpy_wrapper-0.1.188-py3-none-any.whl

Download URL hugpy_wrapper-0.1.188-py3-none-any.whl
Size 512.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8773f99028e1fe8b2d3ec5781c23f286009616b27d02b222e927d8885ead0fb6
BLAKE2b-256 checksum
How to use checksums
ef30c5042c633642d8c25e702df6a31007bc4fbc7faefe8b41227b03cfcae719
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Security for hugpy-wrapper 0.1.188

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.188 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page