gpu-quiescence

In plain English: this checks a GPU really has enough free memory and has settled before you launch a big job, so it does not crash halfway through.

Train on the box you serve from: a VRAM readiness preflight for machines that serve and train on one GPU.

More tools · Working rules

Provenance: this came out of one body of production LLM work, extracted and generalised into a standalone package. First published 2026-08-31.

One machine, two jobs: an inference server holding models warm, and a training run that needs the memory back. Start the trainer cold and it OOMs, thrashes, or silently degrades serving. gpu-quiescence runs a preflight first, and records precisely what was tested - including which resource each stage actually touched.

evict → settle → probe → headroom → launch (reduced priority) → restore

• Evict: ask the inference server to release its models. Ships an Ollama evictor; anything with evict / settled / restore plugs in. The stage owns the wait: settled() is polled against a deadline, because eviction is asynchronous on real servers.
• Settle: free memory (VRAM in GPU mode, via nvidia-smi) must sit inside a variation band before any reading is trusted. A reclaim still in progress shows up as a wide band and keeps the stage waiting. A timeout too short to ever succeed is rejected at construction.
• Probe: allocate, touch, and release one representative buffer. Honesty first: the default allocator is host memory (bytearray). In GPU mode the VRAM readiness signal comes from settle and headroom reading real VRAM numbers; the probe proves the host side, and the report names its allocator so the record says exactly what was tested. To probe VRAM itself, pass an allocator that touches it (for example lambda n: torch.empty(n, dtype=torch.uint8, device='cuda')) with its own label. Probe outcomes are three-valued: succeeded, refused (the allocator said no - the expected failure), or errored (the allocator broke). Both failures gate the launch; the report never disguises one as the other. And success claims only itself: not "memory is contiguous", not "training cannot OOM".
• Headroom: free >= required * 1.10 + 512 MiB, both knobs configurable. The numbers go in the report either way.
• Launch + restore: the job runs as a reduced-priority child, and the inference model is re-warmed after it exits, success or not. The box goes back to serving either way.

Every stage produces a report entry with its observations; a failed stage gates the job instead of raising, and an empty handshake fails because nothing was tested. --json emits the whole readiness report.

Install

pip install "git+https://github.com/m-sanchez/gpu-quiescence@v2.0.0"

# gate a fine-tune on 8 GiB of GPU memory, evicting Ollama first
gpu-quiescence --require-mib 8192 --gpu \
  --ollama http://127.0.0.1:11434 --restore-model llama3 \
  -- python train.py

Not yet on PyPI; the pinned git tag is the supported install and CI proves the built wheel installs, imports, and exposes the command.

Exit codes: 0 ready (and the job succeeded, if one was given) · 1 not ready, or the job failed · 2 usage error - a missing nvidia-smi is a usage problem, never a "not ready" verdict.

Python 3.10+, zero runtime dependencies. System-RAM mode (--system) reads /proc/meminfo on Linux and GlobalMemoryStatusEx on Windows; other platforms need the optional extra (pip install "gpu-quiescence[fallback]", which brings psutil). GPU mode parses nvidia-smi. Develop with pip install -e . pytest and pytest.

Library use

from gpu_quiescence import Handshake, SettleStage, ProbeStage, HeadroomStage, NvidiaSmiProbe

probe = NvidiaSmiProbe()
report = Handshake([
    SettleStage(probe),
    ProbeStage(probe, required_mib=8192),
    HeadroomStage(probe, required_mib=8192),
]).run()

report.ok           # gate decision
report.to_dict()    # what was tested, stage by stage, with observations

The tests are the point