hearth

Deterministic model residency. Keep declared models warm, and tell the truth about which ones are.

Not an inference engine. llama.cpp and vLLM have spent years on kernels, samplers and tokenizers, and none of that is the problem. The problem is that no serving stack will promise a model stays loaded, and none of them can say why one stopped being.

The night this came from

A PIN operator on one rented RTX A6000 stopped answering. Requests hung, then failed. Four pull requests landed against the streaming path in a single evening and none of them were the cause — because the cause was never visible. Three completely different failures were arriving as the same timeout:

• the runtime evicted a model to free VRAM,
• the host detached the GPU and gave it to another tenant,
• a 32B model was simply still loading.

One of those is a capacity problem you own. One is your provider's and no configuration will touch it. One is not a problem at all. A timeout cannot tell you which, so all three got "fixed" repeatedly and none of them went away.

Worse, the operator was being scored down for a card their host reclaimed. A reputation system fed that kind of data slowly deletes its own honest operators.

What it does

$ hearth status
42.0 / 44.2 GiB held  (5 declared, 2 admitted)
  muse-local:latest            resident for 14203s
  deepseek-r1:32b              loading for 47s
  gemma4:26b                   not admitted — short by 14.8 GiB
  qwen3.6:27b                  not admitted — short by 15.8 GiB
  gemma4-extract:31b           not admitted — short by 17.8 GiB

Three things, none of which you can get today:

1. Residency is a named state with a named reason.

Unknown    never probed
Loading    weights materializing — with elapsed, so it can say how long
Resident   loaded AND answering AND accounted for
Lost       with a reason: Evicted · GpuDetached · ProcessExited · Unhealthy
Failed     won't load, and what the runtime said
Stopped    unloaded on purpose

Evicted and GpuDetached are the two states nothing else reports, and they were the two that mattered. They are distinguished by one bit — whether the GPU was still present when the probe failed — and that bit is the entire diagnosis.

2. The card's size is arithmetic, checked before anything loads.

Declare four 20 GiB models on a 48 GiB card and no runtime errors. It loads, evicts, loads, evicts, forever, and presents as "the models got slow." hearth refuses the fifth model at declare time and tells you it was short by 17.8 GiB. Nothing is ever evicted to make room for a load — if it doesn't fit, the honest answer is that it doesn't fit.

3. Routers get an answer they can act on.

Design rules

Nothing fails on a clock. A 32B materializing over a network fabric can legitimately spend minutes before its first token, and killing it at an arbitrary deadline turns a slow success into a fast failure. Loading reports how long it has been loading; deciding what to do about that belongs to whoever knows if a human is waiting. Progress is reported — patience is a policy, not a constant.

Loading is not Ready. The most common way a serving stack lies is routing to something still coming up and calling the inevitable timeout an error.

Declaration order is priority order. First fit, never best fit. Reordering to squeeze in one more model would silently demote whatever the operator listed first, and on a serving box first means most important. A planner that outsmarts the operator surprises them at 3am.

The reserve is never planned into. Weights aren't the whole cost — KV cache grows with context and parallelism, the CUDA context is hundreds of megabytes, and fragmentation is real on a card that's been up for weeks.

Status

hearth-core — state machine, VRAM planner, fleet routing. 33 tests, all pure logic, no GPU required to run them.

Next: supervisor over llama-server children · NVML probe · HTTP surface (OpenAI-compatible + /residency) · CLI · napi + PyO3 bindings · NEDB event log.

That last one is the interesting one. Every state transition becomes an event in NEDB, which is bi-temporal — so "what was resident as of 03:14?" is a real query against a real causal chain. When a model goes cold at 3am you get the answer instead of a theory.

Integration

hearth speaks OpenAI-compatible, so pin-clientd works with it today: set apiMode: "openai" and point inferenceUri at hearth. /residency then adds the truth that the OpenAI shape has no way to express.

Build

cargo test          # 33 tests, no GPU needed
cargo run --example a6000

© Interchained LLC · BUSL-1.1 (converts to Apache-2.0 on 2030-08-27)