hugpy-wrapper (fitevict)
pip install hugpy-wrapper # import fitevict; fitevict-serve
pip install "hugpy-wrapper[transformers]" # the transformers engine (own venv)
hugpy's per-box wrapper: an evict-to-fit front door over llama.cpp (and transformers). Point any OpenAI client's base URL at it; it turns one address into a self-managing model server.
A /v1 call carries only a model name — never a location. llama has no idea
where a gguf lives. fitevict is the layer that does what the call can't: for
each request it resolves the name → gguf path, runs an evict-to-fit plan
against what's resident and what's being called, places the model into llama
(launch/evict), forwards the call, splits the model's <think> out of the
answer, and records the call — raw stamps + engine geometry — to a call log that
every metric derives from.
The package depends on nothing else in hugpy. The decision core is pure stdlib; hardware
facts come from nvidia-smi; model/gguf facts from the files on disk.
Layout
fitevict/
types.py frozen data contract (DeviceBudget, Resident, LoadRequest, FitPlan, ...)
evict.py plan_eviction / sort_key — the victim selector (pure)
plan.py plan_fit — staged decision + quant ladder (pure)
flex.py ctx-band compress + layers-that-fit offload (pure)
host.py EngineHost protocol + drive() loop (measure→plan→evict→load)
front_door.py the WRAPPER: persistent OpenAI /v1 server (owns the address)
adapters/
llama_cpp.py concrete EngineHost: measure GPU/RAM, price GGUFs, launch/evict llama-server
db.py call_log writer (Postgres inference_engine.call_log)
lifted/ measure/act helpers lifted clean from hugpy (imports stripped):
gguf_inspect, gguf_need, hardware, spill_reserve, model_resolve,
native_resolve, supervisor/procutil, timings, no_think, app_dirs, ...
run_engine.py a juggling harness: discover models, fire a sequence that exceeds the card
Install
pip install . # core + wrapper (stdlib only)
pip install .[db] # + psycopg for direct call_log DSN writes (else shells to psql)
pip install .[test] # + pytest
Run
fitevict-serve --host 127.0.0.1 --port 8080 # the front door (owns /v1)
fitevict-run # the juggle harness on this box
Then point any OpenAI client at http://127.0.0.1:8080/v1:
curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"<name>","messages":[{"role":"user","content":"hi"}],"max_tokens":128}'
The model is named, not located; the wrapper finds it, fits it, serves it, logs it.
Every model kind, one front door
Chat (GGUF via llama-server, transformers via tf_child), vision (vision_child)
and every non-chat engine go through the SAME placement (evict-to-fit), the SAME
call_log row and the SAME residents/models catalog. Non-chat engines are runners
on one generic child (lifted/task_child.py); each is one file in
lifted/runners/ plus one RunnerSpec:
| route | runner | serves |
|---|---|---|
POST /v1/embeddings |
embed |
sentence-transformers encoders |
POST /v1/audio/transcriptions, /v1/audio/translations |
asr |
whisper (multipart file or file_path/file_b64) |
POST /v1/audio/speech |
tts |
chatterbox — returns raw audio bytes |
POST /v1/images/generations, /edits, /variations |
imagegen |
diffusers image pipelines — b64_json (or response_format: path) |
POST /v1/videos/generations |
videogen |
diffusers video pipelines (Wan / LTX / CogVideoX / Mochi …) — mp4 path |
POST /v1/summarize |
summarize |
seq2seq summarizers (bart / pegasus / t5 / led) |
POST /v1/keywords |
keywords |
KeyBERT over a stamped embedding model |
Binary outputs are written to FITEVICT_ARTIFACT_DIR and returned in the
endpoint's OpenAI shape; call_log stores their exact bytes + sha256, never the
bytes. Uploads are spooled to FITEVICT_SPOOL_DIR and removed after the call.
models.runner / residents.runner name the engine path of every entry.
Env profiles (lifted/profiles.py): a runner's extra pip deps live in a
per-profile venv at <engine>/envs/<name> — an overlay on the box's worker base
(FITEVICT_TF_PYTHON) or, when a model pins its own torch (chatterbox), an
isolated venv. Declared in runners.PROFILES, overridable in
hugpy_core.env_profiles; a model can be pinned to one with
model_workers.env_profile. A profile is built on first use, off the request
path; until it is ready the call fails as data (503 profile_materializing).
GET /profiles reports every profile's state and which models ride it.
Test
pytest # pure unit tests + a call_log replay + the VL-flood fixture
1 maintainer
Metadata
1 maintainer
Release files for hugpy-wrapper 0.1.7
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| hugpy_wrapper-0.1.7.tar.gz | 250.5 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| hugpy_wrapper-0.1.7-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 492.7 kB
Release files / hugpy_wrapper-0.1.7.tar.gz
| Download URL | hugpy_wrapper-0.1.7.tar.gz |
|---|---|
| Size | 250.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
11e73cb64a631dd4b72f107e246f36dda475d397fd06ddca0a95ab33c849b4c0
|
|
BLAKE2b-256 checksum How to use checksums |
2f909493598479ba68ffc78c6b800ee7c1ac1281fa08b01620347b1cf0ba8e78
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.3
|
Release files / hugpy_wrapper-0.1.7-py3-none-any.whl
| Download URL | hugpy_wrapper-0.1.7-py3-none-any.whl |
|---|---|
| Size | 242.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8675509b426b910932f5cccbb9c5db67eb48ba4508232fdd38109dbc2664bc85
|
|
BLAKE2b-256 checksum How to use checksums |
11a16c7294aca7bb5fc56d8f65866c84cd92fffde39a1b385af5dcc676241d62
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.3
|
