gpu-quiescence

In plain English: this checks a GPU really has enough free memory and has settled before you launch a big job, so it does not crash halfway through.

Train on the box you serve from: a VRAM readiness preflight for machines that serve and train on one GPU.

More tools · Working rules

Provenance: this came out of one body of production LLM work, extracted and generalised into a standalone package. First published 2026-08-31.

One machine, two jobs: an inference server holding models warm, and a training run that needs the memory back. Start the trainer cold and it OOMs, thrashes, or silently degrades serving. gpu-quiescence runs a preflight first, and records precisely what was tested - including which resource each stage actually touched. Every observation carries the resource it measured, and the envelope names the tool and version that produced it, so an archived --json record identifies itself.

evict → settle → probe → headroom → launch (reduced priority) → restore

• Evict: ask the inference server to release its models, then check that the memory actually came back. Ships an Ollama evictor; anything with evict / settled / restore plugs in. settled() is a predicate and the stage owns the only deadline, so the report states the time it really waited and counts every question it asked. /api/ps says how much VRAM each model holds, so the reclaim is certified in bytes: free memory must rise by at least 90% of what the server said it was holding (--reclaim-fraction, 0 to switch the check off). "The server says it unloaded" is a claim; the delta is a measurement.
• Settle: free memory (VRAM in GPU mode, via nvidia-smi) must sit inside a variation band before any reading is trusted. A reclaim still in progress shows up as a wide band and keeps the stage waiting. A timeout too short to ever succeed is rejected at construction.
• Probe: allocate, touch, and release one representative buffer. Honesty first: the default allocator is host memory (bytearray). In GPU mode the VRAM readiness signal comes from settle and headroom reading real VRAM numbers; the probe proves the host side, and the report names its allocator so the record says exactly what was tested. To probe VRAM itself, pass --probe-vram (library: ProbeStage(..., allocator=torch_cuda_allocator())), which allocates and fills a uint8 tensor on the CUDA device and needs the torch extra. Probe outcomes are three-valued: succeeded, refused (the allocator said no - the expected failure), or errored (the allocator broke). A CUDA OOM counts as a refusal even though torch.cuda.OutOfMemoryError is not a MemoryError; refusal_exceptions= teaches the stage about any other allocator. Both failures gate the launch; the report never disguises one as the other. And success claims only itself: not "memory is contiguous", not "training cannot OOM".
• Headroom: free >= required * 1.10 + 512 MiB, both knobs configurable. The numbers go in the report either way.
• Launch + restore: the job runs as a reduced-priority child, and the inference model is re-warmed after it exits, success or not. The box goes back to serving either way.

Every stage produces a report entry with its observations; a failed stage gates the job instead of raising, and an empty handshake fails because nothing was tested. --json emits the whole readiness report.

Install

pip install gpu-quiescence

# gate a fine-tune on 8 GiB of GPU memory, evicting Ollama first
gpu-quiescence --require-mib 8192 --gpu \
  --ollama http://127.0.0.1:11434 --restore-model llama3 \
  -- python train.py

Also installable from a pinned git tag: pip install "git+https://github.com/m-sanchez/gpu-quiescence@v3.0.0". CI proves the built wheel installs, imports, and exposes the command; releases are built and published from a tag by GitHub Actions using PyPI trusted publishing, with PEP 740 attestations, so the wheel can be verified rather than trusted. No required runtime dependencies; psutil and torch are optional extras.

Exit codes: 0 ready (and the job succeeded, if one was given) · 1 not ready, or the job failed · 2 usage error. A usage error is anything that stopped the handshake from running at all - no nvidia-smi, a driver that will not answer, a MIG or vGPU device that reports [N/A] for free memory, an unreachable --ollama URL, a command that does not exist - and it never shares an exit code with a verdict. Every row of that contract has a test against main().

Python 3.10+, zero runtime dependencies. System-RAM mode (--system) reads /proc/meminfo on Linux and GlobalMemoryStatusEx on Windows, and CI runs the suite on both; other platforms need the optional extra (pip install "gpu-quiescence[fallback]", which brings psutil). GPU mode parses nvidia-smi; --probe-vram needs pip install "gpu-quiescence[torch]". Develop with pip install -e . pytest and pytest.

Library use

from gpu_quiescence import Handshake, SettleStage, ProbeStage, HeadroomStage, NvidiaSmiProbe

probe = NvidiaSmiProbe()
report = Handshake([
    SettleStage(probe),
    ProbeStage(probe, required_mib=8192),
    HeadroomStage(probe, required_mib=8192),
]).run()

report.ok           # gate decision
report.to_dict()    # envelope + what was tested, stage by stage

EvictStage(evictor, probe=probe) certifies the reclaim in bytes; ProbeStage(probe, required_mib, allocator=torch_cuda_allocator()) probes VRAM instead of host memory.

The tests are the point

Every externally checkable claim on this page is mapped to the test that enforces it in CLAIMS.md.