Lifeboat Cache

Tiered, compressed, persistent KV cache for vLLM, SGLang, and Lifeboat (which embeds SGLang), with a built-in dashboard to configure, monitor and observe it.

pip install "lifeboatcache[vllm]"   # into the environment that runs vLLM
pip install lifeboatcache           # into the environment that runs SGLang / Lifeboat

Installing registers engine plugins; nothing else is needed to start. With no config file, Lifeboat writes KV to local disk under ~/.lifeboat_cache/kv, compresses it with fp8, and serves the dashboard on http://127.0.0.1:9400.

Turn it on

What happens underneath:

• vLLM (default, engine.vllm_mode: native) — Lifeboat is a secondary tier of vLLM's own KV offloading (OffloadingConnector + TieringOffloadingSpec). vLLM moves GPU ↔ CPU for every KV cache group, including hybrid models' Mamba / linear-attention state; Lifeboat persists those blocks. No LMCache needed.

• vLLM (engine.vllm_mode: lmcache) — LifeboatConnector is LMCache's multiprocess connector with Lifeboat defaults: it auto-starts a local LMCache server (port 6555) whose L2 storage is the Lifeboat adapter. Standard-attention models only.

• SGLang / Lifeboat — Lifeboat is a HiCache storage backend (--hicache-storage-backend dynamic). With LIFEBOAT_CACHE=1 the plugin enables HiCache and selects the backend unless the launch already chose a storage backend. Explicit flags work too:

  python -m sglang.launch_server --model-path <model> --enable-hierarchical-cache \
    --hicache-storage-backend dynamic \
    --hicache-storage-backend-extra-config '{"backend_name":"lifeboat","module_path":"lifeboat_cache.integrations.sglang_backend","class_name":"LifeboatHiCacheStorage"}'
  

Where it runs

Lifeboat Cache plugs into the engine's own KV offload path, so it runs where that path exists. Everywhere else LIFEBOAT_CACHE=1 leaves the launch exactly as it would be without it, and logs why.

Measured on an Apple M4 Max (2026-10-27), Qwen3-0.6B, a 6,123-token prompt asked again after the engine's prefix cache was cleared and after a full restart (same answer each time):

Several users returning at once, measured with the public verification tool (LifeboatCacheTest, lifeboatcache from PyPI, same Mac and model; CSVs in docs/reports/verify-mac-m4max-0.1.14/ (vLLM) and -0.1.15/ (SGLang)):

On MLX, Lifeboat stores full 64-token chunks uncompressed when a request finishes (copied out of the pool on the GPU, written by a background thread), and loads them back when a prefix match stops short. Reads start when a request arrives, so several returning users' reads overlap (0.1.15); the match then writes the chunks into the pool in one update, about 0.3 s per 16K user on this model. 4 users x 16K returning together: 2.0 s to the last user's first token, 2.5-2.6 s without the prefetch. Hybrid (Mamba / sliding-window) models keep SGLang's cache.

vLLM: install and start

# into the Python environment that runs vLLM (vLLM >= 0.32; LMCache not required)
pip install lifeboatcache

# zero-flag: the plugin wires vLLM's native offloading -> CPU tier -> Lifeboat Cache
LIFEBOAT_CACHE=1 vllm serve /models/Qwen3.8-27B --served-model-name my-model \
  --kv-cache-dtype fp8 --enable-prefix-caching

# explicit (same result, no env var)
vllm serve /models/Qwen3.8-27B --served-model-name my-model --enable-prefix-caching \
  --kv-transfer-config '{"kv_connector":"OffloadingConnector","kv_role":"kv_both",
    "kv_connector_extra_config":{"spec_name":"TieringOffloadingSpec","cpu_bytes_to_use":21474836480,
      "secondary_tiers":[{"type":"LifeboatTier","module_path":"lifeboat_cache.integrations.vllm_tier"}]}}'

How it attaches, native mode (no vLLM source changes):

vLLM (GPU KV + Mamba state, prefix cache)
  └─ OffloadingConnector / TieringOffloadingSpec                       [vllm.general_plugins]
       └─ CPU primary tier (/dev/shm, engine.native_cpu_gb, default 20 GB)
            └─ secondary tier: LifeboatTier                 [module_path plugin]
                 └─ tiers: ONTAP / NFS / S3 / local disk (fallback), index, eviction,
                    dashboard :9400, Prometheus :9402

The CPU tier lives in /dev/shm: make sure it has native_cpu_gb free (a vLLM killed with SIGKILL can leave a stale /dev/shm/vllm_offload_*.mmap behind — delete it if nothing maps it). Blocks are stored raw (KV is already fp8 with --kv-cache-dtype fp8; recurrent state must stay lossless), keyed by vLLM's storage namespace (model, parallel layout, dtype, KV groups) so they are never served to an incompatible configuration.

LMCache mode (engine.vllm_mode: lmcache, pip install "lifeboatcache[vllm]", or --kv-transfer-config '{"kv_connector":"LifeboatConnector","kv_role":"kv_both"}'):

vLLM (GPU KV, prefix cache)
  └─ LifeboatConnector  = LMCache MP connector + Lifeboat defaults      [vllm.general_plugins]
       └─ LMCache MP server (auto-started, localhost:6555)  L1 = host RAM
            └─ L2 adapter: LifeboatL2Adapter                 [--l2-adapter plugin]
                 └─ tiers: ONTAP / NFS / S3 / local disk (fallback), compression, index,
                    eviction policy, dashboard :9400, Prometheus :9402

Hybrid (Mamba / linear-attention) models such as Qwen3.5/3.8: use native mode (the default) — 32× faster first token after a restart on Qwen3.8-27B at 256K, see below. Keep --max-num-seqs below the Mamba cache-block count. Under LMCache (0.1.4+) hybrid models restore too, but LMCache's format for them is ≈3.7× larger (≈131 KB/token vs ≈35 KB native), so its 20 GB RAM tier cannot stage a 256K prompt.

Sizing native mode: engine.native_cpu_gb (default 20) is vLLM's host-RAM staging tier; KV restored from disk passes through it, so it bounds how much several returning users can restore at once. vLLM's offloading also allocates ≈19 GB of its own next to it — budget ≈ native_cpu_gb + 20 GB of host RAM. On attention models set engine.native_block_tokens: 256 (vLLM otherwise offloads 16-token blocks).

If vLLM already uses LMCache

Lifeboat Cache never replaces a KV connector you configured — an explicit --kv-transfer-config always wins, LIFEBOAT_CACHE=1 included.

Storage targets and the local fallback

Targets are tried in order; the first usable one takes new writes. Each is write-probed at startup. One that can't be reached (ONTAP/NFS export not mounted, S3 without credentials) is skipped with the reason shown on the dashboard, and if nothing persistent is left, KV goes to storage.fallback_local on local disk.

storage:
  tiers:
    - {name: ontap, type: ontap, enabled: true, mount: /mnt/kvcache, protocol: pnfs_rdma}
    - {name: nfs,   type: nfs,   enabled: true, mount: /mnt/nfs, path: /mnt/nfs/lifeboat/kv}
    - {name: s3,    type: s3,    enabled: true, bucket: kv-archive, region: us-east-1}
  fallback_local: {enabled: true, path: ~/.lifeboat_cache/kv, capacity_gb: 50}

Persisted KV survives restarts: on startup the index is rebuilt from disk in the background.

Several engine processes on one node (one vLLM per GPU behind a load balancer) can share one tier directory: give them the same path and set storage.shared_lookup: true, and a block one copy wrote is found by the others (a user returning on a different GPU is restored, not recomputed). Size capacity_gb for the whole directory; see G23 in docs/KNOWN_GAPS.md.

Dashboard

http://127.0.0.1:9400 on the node (use ssh -L 9400:localhost:9400 <node> from your laptop). It shows the cache hit rate, KV served, bytes saved by compression and the codec's sampled reconstruction error, per-target status (circuit breaker, usage, queue), evictions, and startup recovery. The Configure tab edits storage targets, the local fallback, the compression codec per rung, the eviction policy and the write mode; changes are validated, applied live, and saved to the config file (startup-bound sections are flagged "restart required"). /metrics serves Prometheus text (also on port 9402).

To expose it beyond localhost, set dashboard.host: 0.0.0.0 and dashboard.auth_token; open the page once with ?token=<token>.

CLI: lifeboat-cache status prints the same numbers; lifeboat-cache init writes a starter config; lifeboat-cache server runs a standalone Lifeboat-backed LMCache server for several vLLM instances to share.

Configuration

One YAML file, resolved as: --config / $LIFEBOAT_CACHE_CONFIG → ~/.lifeboat_cache/lifeboat.yaml → built-in defaults ($LIFEBOAT_CACHE_HOME moves the home directory). lifeboat-cache config show prints the resolved config. Presets ship with the package: extends: single_node_gpu.

Measured results (vLLM 0.32 / SGLang 0.5.19, 1× RTX PRO 6000, FP8 KV)

Time to first token for the same prompt after a full server restart (real source-code prompts):

Full benchmark (revision 2, 2026-10-25, docs/reports/Lifeboat_Cache_Benchmark_Report.pdf) — six configurations including vLLM's and LMCache's own disk tiers as baselines, 1/4/16 concurrent users, 1K–256K context, page cache dropped before every disk read, same NVMe and 250 GB budget for every disk tier:

The two tables above were measured with 0.1.4. At that point, as LMCache's disk tier Lifeboat restored 1.4–1.7× faster than LMCache's own fs adapter, and as vLLM's offload tier it was 1.3–2.0× slower than vLLM's own fs tier. 0.1.5–0.1.6 rebuilt the read and write paths (tables below). The large gains over "vLLM only" come from persistence itself, which both engines already ship.

End-to-end re-run (0.1.6–0.1.8, 2026-10-25). Same harness, host, GPU, prompts and 250 GB budget as revision 2; only the Lifeboat columns were re-run (vLLM 0.32 and LMCache 0.5.6 are unchanged), with the cells the tables above quote. Time to first token in seconds, share of the prompt served from cache in brackets where it is not ~100% (docs/reports/e2e_0.1.6.txt):

• vs LMCache's disk tier (LMCache mode): restarts 1.6–2.35× faster on the regular model and 1.4–1.85× on the hybrid.
• vs vLLM's disk tier (native mode): restarts went from up to 2× slower (0.1.4) to level or up to 11% faster (5–11% at 128K–256K). Most of the remaining time is GPU work both share, so faster disk reads show up only partly in TTFT.
• Returning users on the regular model:
  • 16 users × 32K: 2.07× faster than vLLM's fs tier (1.50 s vs 3.11 s; 99% vs 93% of the prompt from cache).
  • 4 users × 128K: level (0.99 s vs 0.98 s); 0.1.4–0.1.7 recomputed one user (24.6 s).
  • How:
    • 0.1.7: lookups are asynchronous, through vLLM's own file-tier protocol.
    • 0.1.8: when vLLM's CPU tier is momentarily full, a request whose blocks are on disk waits for room (up to engine.native_promotion_wait_s, 5 s) instead of being recomputed. vLLM's fs tier recomputes it.
• Hybrid on LMCache stays unsupported at 256K (LMCache doesn't pass the recurrent state); use native mode.

CPU-tier size (0.1.8, 2026-10-26). The overflow cells re-run with vLLM's CPU tier (engine.native_cpu_gb) at 20 GB and 30 GB, the most a 64 GB host can spare next to the engine, and with engine.native_promotion_wait_s at 5 s and 30 s. Returning users, mean TTFT in seconds (share from cache); – = not run (docs/reports/ram_tier_0.1.8.txt):

• 30 GB fixes 16 users × 64K: 3.40 s, 3.2× faster than vLLM's fs tier given the same 30 GB.
• 4 users × 256K is about 51 GB of KV and doesn't fit 30 GB: one user is recomputed in every configuration. Lifeboat is still 1.2× faster than vLLM fs at 30 GB (76.66 s vs 93.00 s). Fitting all four needs a tier of about 55 GB, i.e. more than 64 GB of RAM.
• Keep the 5 s default wait. 30 s helps only when everything fits (2.93 s vs 3.40 s). When the tier can't hold everyone, the request is recomputed anyway, just later (31.41 s vs 12.57 s at 20 GB; 122.34 s vs 76.66 s at 30 GB).
• Hybrid model at 30 GB: 16 × 64K is 3.6× faster than vLLM fs at the same 30 GB (5.23 s vs 18.86 s). At 4 × 256K both fit at 30 GB and are level (2.77 s vs 2.83 s): that gain over 20 GB (56.52 s) comes from the bigger tier, not from Lifeboat.
• Single-user and restart times don't depend on tier size and were unchanged.

Storage layer head-to-head (0.1.6, 2026-10-25). Both adapters were driven through LMCache's L2 interface on the same host and NVMe (Intel Core Ultra 9 285K; 6.9 GB/s O_DIRECT, 5.3 GB/s buffered). The data was 6.4 GB of Qwen3-30B-A3B-shaped chunks (48 layers × 512 × 256 tokens), read 32 chunks per request. "Cold" means the page cache was dropped before reading; "hot" means just written. Throughput is in GB/s of KV restored:

¹ Measured before the store path lost its extra copies; only the FP8, 1-request row was re-run after. Store runs vary ±30% with page-cache writeback from earlier runs.

• One request: 2.0–2.2× faster than LMCache's default fs adapter, and 14–16% faster than LMCache with O_DIRECT.
• 4 concurrent requests: 1.4× faster than the default adapter. LMCache with O_DIRECT ties, because both run at the drive's limit.
• Just-written KV: 1.4–3.1× faster.
• Stores: 1.8–2.1× faster (FP8, re-measured).
• Limits:
  • At the drive's limit, only reading fewer bytes goes faster.
  • A block read with O_DIRECT is not left in the page cache. A repeat read that LMCache's buffered adapter serves from RAM (13.6–17.3 GB/s at 4 requests) runs at disk speed here. direct_io: false on a tier restores page-cache behavior.

Native vLLM tier, same host, 1 GB of blocks, cold: 1.38 GB/s (0.1.4) → 4.04 (0.1.5) → 6.3–6.4 GB/s (0.1.6). vLLM's own fs tier measured 3.2 GB/s there.

What changed in 0.1.6:

• Cold reads go O_DIRECT. New files align the payload to 4 KiB for this, and a read is bounced through an aligned buffer when the engine's buffer isn't 64-byte aligned.
• Blocks already in the page cache are served from it first (RWF_NOWAIT).
• Uncompressed chunks are read straight into LMCache's buffers.
• Store and load threads hash single-threaded. BLAKE3's own thread pool oversubscribed the cores (6–9 ms vs 0.7 ms per chunk).
• Stores encode, hash and write each chunk on its own worker, with one copy instead of three.

Lifeboat's other additions are operational: capacity limit, free-disk reserve, integrity, indexed manifest, ONTAP/NFS/S3 fallback, SGLang and the dashboard.

Storage on that host: Lifeboat restore 4.8–5.2 GB/s from NVMe (raw device 6.7 GB/s), 17–19 GB/s from RAM (above a 100 Gb/s RDMA link), GPU H2D 28.7 GB/s; recovery of 100K blocks from the SQLite manifest 0.07 s vs 4.8 s by directory scan.

Qwen3-30B-A3B rows use LMCache mode; the native rows restored 95/95 blocks (4.9 GB) with a 95.5% external prefix-cache hit rate; greedy outputs matched the pre-restart outputs in 3/4 prompts (the 4th diverged mid-sentence — vLLM with fp8 KV is not bit-deterministic even without a cache).

• 99.7% of the prompt's KV served from Lifeboat after restart on the standard-attention model.
• Cold-request overhead ≈1% (LMCache mode); native mode +0.1–2% at 32K–128K, +9% at 8K (1.37 s vs 1.25 s); GPU-warm repeats unchanged.
• Restore throughput 12.2 GiB/s (zero-copy mmap, parallel BLAKE3, one copy); index rebuild for 16 GB of KV in 0.06 s.
• Hybrid (Mamba / linear-attention) models: on SGLang Lifeboat persists the attention KV and the recurrent state (HiCache v2 sidecar pools, state stored losslessly) → 1.8–5.8× after restart. On vLLM, native mode stores both through vLLM's own offloading → 5.3–32× after restart. Under LMCache they restore from 0.1.4 on (0.1.3 failed every hybrid restore), but LMCache's 20 GB L1 cannot stage a 256K hybrid prompt — prefer native mode for hybrid models.
• After a long prompt, allow SGLang's write-through (~30 s for 128K tokens) to finish before a restart; an unfinished tail is recomputed after the restart, never corrupted.

Compression — what the numbers say

The lossy codecs apply to 16-bit KV. fp8 is the default for every stored rung. The dashboard shows the sampled reconstruction error of whatever codec is live.

Lossless (lossless, type zkv, 0.1.6). It splits each element's bytes into planes and codes each plane with zstd. Decoding is exact.

• fp8 KV caches (--kv-cache-dtype fp8) can't be shrunk by the lossy codecs.
  • On shared and archive rungs they are coded losslessly. Those links are slower than decode.
  • On local NVMe they are stored raw. Decoding there cost more time than the 21% of bytes it saved (cold 4.6–5.1 vs 6.4–6.8 GB/s).
  • compression.lossless_8bit: on | remote | off controls this.
• max_accuracy uses lossless on its local rungs.

LBQ (0.1.5) is Lifeboat's low-bit codec. For each stored chunk it:

• centers the keys per chunk (attention ignores a shared key offset);
• applies a randomized Hadamard rotation;
• stores a per-vector fp16 scale and codes values with a Lloyd-Max codebook;
• uses separate key / value bit widths;
• keeps the first / last layer and the chunk that opens a sequence (the attention sink) at fp8.

Qwen3-4B, BF16 KV. Code and prose prompts are stored and restored through the live path in 256-token chunks, then a 512-token continuation is scored against the uncompressed cache (benchmarks/accuracy/codec_accuracy.py, docs/reports/codec_accuracy_qwen3-4b.json):

The old algorithms on Qwen3-30B-A3B (2K / 8K / 24K): turboquant4 2.7× at +0–4% perplexity; turboquant3 +44–64%; 2-bit OSCAR per sequence +955–1,379%. Per stored chunk, OSCAR compressed nothing (1.0×). LBQ has not yet been measured on the 30B model.

When to use which:

• LBQ decodes slower than fp8. Per 25 MB chunk (raw-equivalent): ~1 GB/s vs 4.3 GB/s for fp8 on eqprod's x86 CPU with 8 threads, and ~1.5 GB/s vs ~10 GB/s on an Apple M4 Max. Use it where bytes cost more than CPU: shared / archive rungs and capacity-bound disks.
• fp8 stays the default for local NVMe, where restore time matters most.
• storage_efficient uses lbq3 everywhere.