hearth-llm 0.1.0

Keeps your local LLM contexts warm: persistent KV-cache proxy for llama-server

hearth 🔥

Keeps your local LLM contexts warm.

hearth is a zero-dependency caching proxy for llama.cpp's llama-server that persists KV-caches to disk and restores them across sessions — and across server restarts.

The pain point

Local inference has a cold-context problem. Every time you start a new session with llama.cpp or anything built on it, your entire context — the codebase you loaded, the contract you're analyzing, the long conversation you were having — is re-processed token by token from scratch. On consumer hardware that's seconds to minutes of prompt processing, thrown away the moment the process exits.

The KV-cache that represents all that work is just memory. hearth snapshots it to disk, indexes it by conversation content, and restores the longest matching prefix automatically on your next request. Quit, reboot, come back tomorrow — your context is still warm.

How it works

your app (any OpenAI client)
        │
        ▼
   hearth proxy  ──────────►  llama-server
   :8737                      :8738
        │                        │
        ▼                        ▼
   manifest.json  ◄────────  *.kv snapshots
        (~/.hearth/cache — shared directory)

1. Each chat request's messages are hashed as a rolling prefix chain (one digest per message boundary, seeded by model id).
2. hearth finds the longest saved prefix and tells llama-server to restore that snapshot into the slot via its native /slots/{id}?action=restore API. llama-server's own token-level prefix reuse then skips everything already computed.
3. The request is forwarded unchanged (streaming included) and the assistant's reply is captured on the way through.
4. After the response, the slot's new KV state is saved under the hash of the extended conversation — ready for the next turn, or the next restart.

Snapshots are evicted LRU under a configurable size cap. Nothing about your client changes: it's the same OpenAI-compatible endpoint, plus two response headers (X-Hearth-Cache: hit|miss, X-Hearth-Restored-Msgs).

Install

Requires Python ≥ 3.10 and llama-server on your PATH (brew install llama.cpp). hearth itself has zero dependencies.

pip install hearth-llm        # or: pipx install hearth-llm
# or run straight from a checkout — it's stdlib-only:
python3 -m hearth --help

Quickstart

# Easiest: reuse a model you already pulled with Ollama
hearth serve --ollama gemma4:e2b

# Or any GGUF file
hearth serve --model ~/models/qwen2.5-7b-instruct-q4_k_m.gguf

# Or attach to a llama-server you manage yourself
# (it must run with --slot-save-path ~/.hearth/cache)
hearth serve --upstream http://127.0.0.1:8080

Then point any OpenAI client at http://127.0.0.1:8737/v1.

hearth models        # local Ollama models hearth can serve
hearth ls            # saved snapshots (size, tokens, hits, age)
hearth rm <key>      # drop one snapshot
hearth rm --all      # clear the cache
curl localhost:8737/hearth/stats   # hit/miss counters

Benchmark

scripts/bench.py simulates the real workflow: load a large document, ask a question, kill the server entirely, restart, ask a follow-up.

python3 scripts/bench.py --ollama gemma4:e2b

Cold = full prompt re-processing. Warm = hearth restores the snapshot from disk. Measured on an Apple M3 Max (llama.cpp b10090):

	cold	warm
prompt tokens computed	1,333	59
prompt processing time	209ms	6ms

The warm session — a brand-new llama-server process — skipped 96% of prompt processing because the KV state came off disk. The absolute savings scale with model and context size: on a 7B model with a 50k-token codebase loaded, that difference is minutes, not milliseconds.

Note: Ollama blobs for multimodal models (e.g. Gemma) bundle vision tensors that plain llama-server can't load; use a text-only GGUF for those. Text-only Ollama models work directly via --ollama.

Status / roadmap

MVP. Single slot, requests serialized. Planned:

• multi-slot scheduling (parallel conversations, slot affinity)
• snapshot forking (branch a conversation from any saved prefix)
• token-level (not message-level) prefix matching via /tokenize
• cross-machine snapshot sharing (same model + build)
• hearth warm <file> — pre-bake a document/codebase into a snapshot
• TTL-based eviction policies alongside LRU

Development

python3 -m unittest discover -s tests   # no model needed; uses a fake llama-server

See CONTRIBUTING.md — benchmark reports from different hardware are especially welcome.

License

MIT © 2026 Daniel Soromou