hearth 🔥

Keeps your local LLM contexts warm.

Ask a follow-up question about a 30k-token document after a full llama-server restart: 99 seconds cold, 1.8 seconds with hearth — because the KV-cache came off disk instead of being recomputed. Benchmarks below.

hearth is a zero-dependency caching proxy for llama.cpp's llama-server that persists KV-caches to disk and restores them across sessions — and across server restarts.

The pain point

Local inference has a cold-context problem. Every time you start a new session with llama.cpp or anything built on it, your entire context — the codebase you loaded, the contract you're analyzing, the long conversation you were having — is re-processed token by token from scratch. On consumer hardware that's seconds to minutes of prompt processing, thrown away the moment the process exits.

The KV-cache that represents all that work is just memory. hearth snapshots it to disk, indexes it by conversation content, and restores the longest matching prefix automatically on your next request. Quit, reboot, come back tomorrow — your context is still warm.

How it works

your app (any OpenAI client)
        │
        ▼
   hearth proxy  ──────────►  llama-server
   :8737                      :8738
        │                        │
        ▼                        ▼
   manifest.json  ◄────────  *.kv snapshots
        (~/.hearth/cache — shared directory)

1. Each chat request's messages are hashed as a prefix hash chain (one digest per message boundary, seeded by model id).
2. hearth finds the longest saved prefix and tells llama-server to restore that snapshot into its sequence slot — llama.cpp's per-conversation container of KV-cache state — via its native /slots/{id}?action=restore API. llama-server's own token-level prefix reuse then skips everything already computed.
3. The request is forwarded unchanged (streaming included) and the assistant's reply is captured on the way through.
4. After the response, the slot's new KV state is saved under the hash of the extended conversation — ready for the next turn, or the next restart.

Snapshots are evicted least-recently-used (LRU) under a configurable size cap. Nothing about your client changes: it's the same OpenAI-compatible endpoint, plus a few response headers: X-Hearth-Cache: hit|miss, X-Hearth-Restored-Msgs (leading messages the restored snapshot covered), X-Hearth-Restored-Tokens (prefill tokens skipped), and on cache hits X-Hearth-Est-Saved-Ms (present once a few misses have calibrated the baseline prefill rate).

Security & privacy

KV snapshots are equivalent to a plaintext transcript of your conversation: anyone with read access to the cache directory and knowledge of the model can reconstruct the context. hearth stores them unencrypted in ~/.hearth/cache (override with --cache-dir). If you work with sensitive documents:

• rely on full-disk encryption, or point --cache-dir at an encrypted volume;
• run hearth rm --all when you are done with a sensitive context;
• note that snapshots persist until LRU eviction (default 20 GB cap) — deleting the chat in your client does not delete the snapshot.

The proxy and the spawned llama-server bind 127.0.0.1 and have no authentication; do not expose either with --host 0.0.0.0 on an untrusted network. See SECURITY.md for reporting.

Install

pip install hearth-llm        # or: pipx install hearth-llm
# or run straight from a checkout — it's stdlib-only:
python3 -m hearth --help

Requirements

• Python ≥ 3.10, nothing else — hearth is stdlib-only.
• llama-server with slot save/restore (/slots/{id}?action=save|restore), started with --slot-save-path (brew install llama.cpp). hearth passes the flag itself in --model/--ollama mode. Tested against llama.cpp b10090; on builds without the slots API hearth stays functional but logs save/restore failures and every request pays full prefill.
• Context window: the spawned server defaults to --ctx 8192. Raise it to fit your documents, e.g. --ctx 32768 for a 30k-token context.
• Text-only GGUFs for --ollama: Ollama blobs for multimodal models (e.g. Gemma vision variants) bundle tensors plain llama-server can't load; pick a text-only model, or pass a text-only GGUF via --model.

Quickstart

# Easiest: reuse a text-only model you already pulled with Ollama
hearth serve --ollama qwen2.5:7b

# Or any GGUF file
hearth serve --model ~/models/qwen2.5-7b-instruct-q4_k_m.gguf

# Or attach to a llama-server you manage yourself
# (it must run with --slot-save-path ~/.hearth/cache)
hearth serve --upstream http://127.0.0.1:8080

Then point any OpenAI client at http://127.0.0.1:8737/v1.

hearth models        # local Ollama models hearth can serve
hearth ls            # saved snapshots (size, tokens, hits, age)
hearth rm <key>      # drop one snapshot
hearth rm --all      # clear the cache
hearth stats         # what hearth is saving you, live

hearth stats keeps a running comparison of prompt-processing time with and without hearth; the without-hearth baseline is estimated from the prefill rate observed on cache misses:

hearth session stats
  requests                  2   (hits 1, misses 1 — 50% hit rate)
  prompt tokens    computed      2,115
                   restored      2,125   (skipped, served from disk)
  prompt time      with hearth         3.6 s
                   without hearth      7.2 s   (est.)
  saved                 3.6 s   (~2.0x faster prompt processing)
  snapshots                 1   (116.2 MB)

The same numbers are on every chat response via the X-Hearth-* headers and as JSON at GET /hearth/stats.

Configuration

hearth serve flags:

flag	default	meaning
--port	8737	proxy listen port (point clients here)
--host	127.0.0.1	proxy bind address (see Security & privacy before changing)
--upstream-port	8738	port for the spawned llama-server
--ctx	8192	context size passed to the spawned llama-server
--cache-dir	~/.hearth/cache	snapshots + manifest; in --upstream mode must equal the server's --slot-save-path
--max-cache-gb	20	LRU eviction cap
trailing args	—	passed through to llama-server; dash-prefixed args need a -- separator: hearth serve --model m.gguf -- --top-k 4

hearth stats takes --url (default http://127.0.0.1:8737); hearth ls / hearth rm take --cache-dir.

Benchmark

scripts/bench.py simulates the real workflow: load a large document, ask a question, kill the server entirely, restart, ask a follow-up.

python3 scripts/bench.py --ollama qwen2.5:7b

Cold = full prompt re-processing in a fresh llama-server process. Warm = an equally fresh process, but hearth restores the KV snapshot from disk. Measured with Qwen2.5-7B-Instruct Q4_K_M on an Apple M3 Max (llama.cpp b10090), mean of 3 runs each:

context size	cold: time to answer	warm: time to answer	prompt tokens computed (cold → warm)	speedup
~3.7k tok	6.8 s	0.55 s	3,732 → 16	12×
~15.3k tok	34.0 s	1.01 s	15,330 → 16	34×
~30.3k tok	99.0 s	1.79 s	30,294 → 16	55×

At 30k tokens of context — a decent-sized codebase or a long contract — the follow-up question that took 99 seconds cold takes 1.8 seconds warm, and the answers are identical in substance: the model answered document questions correctly from the restored state in every run. The gap widens with context length; the cost of a cache hit is essentially just reading the snapshot back from local storage.

Memory & storage cost

context size	snapshot on disk	KV per token	save (after response)	restore (on warm path)	restore throughput	llama-server RSS
~3.7k tok	0.20 GB	56.9 KB	0.26 s	0.04 s	5.0 GB/s	5.3 GB
~15.3k tok	0.82 GB	56.3 KB	0.95 s	0.19 s	4.4 GB/s	5.4 GB
~30.3k tok	1.62 GB	56.2 KB	2.04 s	0.46 s	3.5 GB/s	6.2 GB

The trade in one sentence: keeping a 30k-token context warm without hearth means keeping a 6.2 GB llama-server process resident forever; with hearth it costs 1.6 GB of idle disk and a half-second restore, and your RAM comes back between sessions. Snapshot size matches the analytic KV footprint for this architecture (2 × 28 layers × 4 KV heads × 128 dims × 2 bytes = 56 KB/token) — hearth adds only a JSON manifest. Saves happen after the response is delivered, so the user never waits on them; hearth's own proxy process stays under 100 MB.

Across models

The same experiment at the ~15k-token context, across six models from five families (mean of 3 runs; token counts differ by tokenizer):

model	params	KV/token (analytic → measured)	cold	warm	speedup	snapshot
Qwen2.5-1.5B	1.5B	28 → 28.1 KB	8.8 s	0.30 s	29×	0.41 GB
Gemma-4-E2B	E2B	iSWA → 6.4 KB	13.8 s	12.08 s	1.1×	0.09 GB
Phi-4-mini	3.8B	128 → 128.4 KB	18.7 s	0.77 s	24×	1.68 GB
Llama-3.2-3B	3B	112 → 112.5 KB	15.8 s	0.81 s	20×	1.47 GB
Mistral-7B-v0.3	7B	128 → 128.5 KB	47.3 s	1.67 s	28×	2.06 GB
Qwen2.5-7B	7B	56 → 56.3 KB	39.0 s	1.23 s	32×	0.82 GB

Three things worth noticing. Measured snapshot sizes match the analytic KV formula (2 × layers × KV heads × head dim × 2 bytes) within 0.5% on every dense-attention model — disk cost is set by attention geometry, not parameter count (Phi-4-mini stores the same 128 KB/token as Mistral-7B at half the size). Gemma-4-E2B is the boundary condition: its interleaved sliding-window attention makes snapshots ~20× smaller, but llama.cpp can't reuse a restored SWA prefix past a divergence point, so the warm run recomputes the prompt and hearth only helps ~1.1× — restores succeed and answers stay correct, but the benefit is engine-limited. And warm answers equal cold answers on every model including the mistakes (verified with a full-prefill control): restored state reproduces model behavior exactly.

Reproduce with:

python3 scripts/bench.py --model <model.gguf> --sweep 60,240,470 --repeat 3
# sweep = document chunks of ~62 tokens: 60 ≈ 3.7k, 240 ≈ 15.3k, 470 ≈ 30.3k tokens of context

# cross-model suite (one model at a time, reclaims snapshot disk between models)
python3 scripts/bench_suite.py --out /tmp/xbench --paragraphs 240 --repeat 3 model1.gguf model2.gguf ...

Methodology notes. Restore throughput is measured with a warm OS page cache — the common quit-and-return case — so it reflects the storage hierarchy (SSD + page cache), not guaranteed cold-device reads; a restore right after a reboot will be somewhat slower. End-to-end speedups depend on the benchmark's 48-token answer cap: decode cost is identical with and without hearth, so longer answers dilute the ratio (a 500-token answer turns the 55× into roughly 9×). The length-independent comparison is the prefill phase itself: ~98 s of cold prompt processing vs a 0.46 s restore. Correctness is a functional sanity check (one deterministic document question per context size at temperature 0), not a semantic evaluation.

To cite hearth, use the repository's CITATION.cff (GitHub's "Cite this repository" button).

Troubleshooting

• Address already in use — something else is on 8737 (proxy) or 8738 (spawned llama-server); change with --port / --upstream-port.
• llama-server exited with code N — its output is captured in <cache-dir>/llama-server.log (default ~/.hearth/cache/llama-server.log); usual causes are a bad GGUF path, a multimodal blob, or too little memory for model + --ctx.
• Everything is a miss after upgrading llama.cpp — restored state is validated by the engine against the exact model/build/context configuration; on mismatch hearth logs restore failed, treating as miss, drops the entry, and falls back to full prefill (correctness is never at risk). Reclaim disk with hearth rm --all after an engine or quantization change.
• Attach mode (--upstream) never hits the cache — the external llama-server's --slot-save-path must be exactly hearth's --cache-dir, and it should run with --parallel 1 (hearth drives slot 0 only). Check hearth's log for save/restore failures.
• Edited an earlier message and lost the cache — matching is message-granular by design; editing any earlier message forfeits reuse from that point (token-level matching is on the roadmap).

Status / roadmap

MVP. Single slot, requests serialized. Planned:

• multi-slot scheduling (parallel conversations, slot affinity)
• snapshot forking (branch a conversation from any saved prefix)
• token-level (not message-level) prefix matching via /tokenize
• cross-machine snapshot sharing (same model + build)
• hearth warm <file> — pre-bake a document/codebase into a snapshot
• TTL-based eviction policies alongside LRU

Development

python3 -m unittest discover -s tests   # no model needed; uses a fake llama-server

See CONTRIBUTING.md — benchmark reports from different hardware are especially welcome.

License

MIT © 2026 Daniel Soromou