Lifeboat Cache

Tiered, compressed, persistent KV cache for vLLM, SGLang, and Lifeboat (which embeds SGLang), with a built-in dashboard to configure, monitor and observe it.

pip install "lifeboatcache[vllm]"   # into the environment that runs vLLM
pip install lifeboatcache           # into the environment that runs SGLang / Lifeboat

Installing registers engine plugins; nothing else is needed to start. With no config file, Lifeboat writes KV to local disk under ~/.lifeboat_cache/kv, compresses it with fp8, and serves the dashboard on http://127.0.0.1:9400.

Turn it on

What happens underneath:

• vLLM (default, engine.vllm_mode: native) — Lifeboat is a secondary tier of vLLM's own KV offloading (OffloadingConnector + TieringOffloadingSpec). vLLM moves GPU ↔ CPU for every KV cache group, including hybrid models' Mamba / linear-attention state; Lifeboat persists those blocks. No LMCache needed.

• vLLM (engine.vllm_mode: lmcache) — LifeboatConnector is LMCache's multiprocess connector with Lifeboat defaults: it auto-starts a local LMCache server (port 6555) whose L2 storage is the Lifeboat adapter. Standard-attention models only.

• SGLang / Lifeboat — Lifeboat is a HiCache storage backend (--hicache-storage-backend dynamic). With LIFEBOAT_CACHE=1 the plugin enables HiCache and selects the backend unless the launch already chose a storage backend. Explicit flags work too:

  python -m sglang.launch_server --model-path <model> --enable-hierarchical-cache \
    --hicache-storage-backend dynamic \
    --hicache-storage-backend-extra-config '{"backend_name":"lifeboat","module_path":"lifeboat_cache.integrations.sglang_backend","class_name":"LifeboatHiCacheStorage"}'
  

vLLM: install and start

# into the Python environment that runs vLLM (vLLM >= 0.32; LMCache not required)
pip install lifeboatcache

# zero-flag: the plugin wires vLLM's native offloading -> CPU tier -> Lifeboat Cache
LIFEBOAT_CACHE=1 vllm serve /models/Qwen3.8-27B --served-model-name my-model \
  --kv-cache-dtype fp8 --enable-prefix-caching

# explicit (same result, no env var)
vllm serve /models/Qwen3.8-27B --served-model-name my-model --enable-prefix-caching \
  --kv-transfer-config '{"kv_connector":"OffloadingConnector","kv_role":"kv_both",
    "kv_connector_extra_config":{"spec_name":"TieringOffloadingSpec","cpu_bytes_to_use":21474836480,
      "secondary_tiers":[{"type":"LifeboatTier","module_path":"lifeboat_cache.integrations.vllm_tier"}]}}'

How it attaches, native mode (no vLLM source changes):

vLLM (GPU KV + Mamba state, prefix cache)
  └─ OffloadingConnector / TieringOffloadingSpec                       [vllm.general_plugins]
       └─ CPU primary tier (/dev/shm, engine.native_cpu_gb, default 20 GB)
            └─ secondary tier: LifeboatTier                 [module_path plugin]
                 └─ tiers: ONTAP / NFS / S3 / local disk (fallback), index, eviction,
                    dashboard :9400, Prometheus :9402

The CPU tier lives in /dev/shm: make sure it has native_cpu_gb free (a vLLM killed with SIGKILL can leave a stale /dev/shm/vllm_offload_*.mmap behind — delete it if nothing maps it). Blocks are stored raw (KV is already fp8 with --kv-cache-dtype fp8; recurrent state must stay lossless), keyed by vLLM's storage namespace (model, parallel layout, dtype, KV groups) so they are never served to an incompatible configuration.

LMCache mode (engine.vllm_mode: lmcache, pip install "lifeboatcache[vllm]", or --kv-transfer-config '{"kv_connector":"LifeboatConnector","kv_role":"kv_both"}'):

vLLM (GPU KV, prefix cache)
  └─ LifeboatConnector  = LMCache MP connector + Lifeboat defaults      [vllm.general_plugins]
       └─ LMCache MP server (auto-started, localhost:6555)  L1 = host RAM
            └─ L2 adapter: LifeboatL2Adapter                 [--l2-adapter plugin]
                 └─ tiers: ONTAP / NFS / S3 / local disk (fallback), compression, index,
                    eviction policy, dashboard :9400, Prometheus :9402

Hybrid (Mamba / linear-attention) models such as Qwen3.5/3.8: use native mode (the default) — 32× faster first token after a restart on Qwen3.8-27B at 256K, see below. Keep --max-num-seqs below the Mamba cache-block count. Under LMCache (0.1.4+) hybrid models restore too, but LMCache's format for them is ≈3.7× larger (≈131 KB/token vs ≈35 KB native), so its 20 GB RAM tier cannot stage a 256K prompt.

Sizing native mode: engine.native_cpu_gb (default 20) is vLLM's host-RAM staging tier; KV restored from disk passes through it, so it bounds how much several returning users can restore at once. vLLM's offloading also allocates ≈19 GB of its own next to it — budget ≈ native_cpu_gb + 20 GB of host RAM. On attention models set engine.native_block_tokens: 256 (vLLM otherwise offloads 16-token blocks).

If vLLM already uses LMCache

Lifeboat Cache never replaces a KV connector you configured — an explicit --kv-transfer-config always wins, LIFEBOAT_CACHE=1 included.

Storage targets and the local fallback

Targets are tried in order; the first usable one takes new writes. Each is write-probed at startup. One that can't be reached (ONTAP/NFS export not mounted, S3 without credentials) is skipped with the reason shown on the dashboard, and if nothing persistent is left, KV goes to storage.fallback_local on local disk.

storage:
  tiers:
    - {name: ontap, type: ontap, enabled: true, mount: /mnt/kvcache, protocol: pnfs_rdma}
    - {name: nfs,   type: nfs,   enabled: true, mount: /mnt/nfs, path: /mnt/nfs/lifeboat/kv}
    - {name: s3,    type: s3,    enabled: true, bucket: kv-archive, region: us-east-1}
  fallback_local: {enabled: true, path: ~/.lifeboat_cache/kv, capacity_gb: 50}

Persisted KV survives restarts: on startup the index is rebuilt from disk in the background.

Dashboard

http://127.0.0.1:9400 on the node (use ssh -L 9400:localhost:9400 <node> from your laptop). It shows the cache hit rate, KV served, bytes saved by compression and the codec's sampled reconstruction error, per-target status (circuit breaker, usage, queue), evictions, and startup recovery. The Configure tab edits storage targets, the local fallback, the compression codec per rung, the eviction policy and the write mode; changes are validated, applied live, and saved to the config file (startup-bound sections are flagged "restart required"). /metrics serves Prometheus text (also on port 9402).

To expose it beyond localhost, set dashboard.host: 0.0.0.0 and dashboard.auth_token; open the page once with ?token=<token>.

CLI: lifeboat-cache status prints the same numbers; lifeboat-cache init writes a starter config; lifeboat-cache server runs a standalone Lifeboat-backed LMCache server for several vLLM instances to share.

Configuration

One YAML file, resolved as: --config / $LIFEBOAT_CACHE_CONFIG → ~/.lifeboat_cache/lifeboat.yaml → built-in defaults ($LIFEBOAT_CACHE_HOME moves the home directory). lifeboat-cache config show prints the resolved config. Presets ship with the package: extends: single_node_gpu.

Measured results (vLLM 0.32 / SGLang 0.5.19, 1× RTX PRO 6000, FP8 KV)

Time to first token for the same prompt after a full server restart (real source-code prompts):

Full benchmark (revision 2, 2026-10-25, docs/reports/Lifeboat_Cache_Benchmark_Report.pdf) — six configurations including vLLM's and LMCache's own disk tiers as baselines, 1/4/16 concurrent users, 1K–256K context, page cache dropped before every disk read, same NVMe and 250 GB budget for every disk tier:

Where Lifeboat stands: as LMCache's disk tier it restores 1.4–1.7× faster than LMCache's own fs adapter; as vLLM's offload tier it is 1.3–2.0× slower than vLLM's own fs tier (its load jobs read blocks sequentially with a BLAKE3 check per block — parallel reads are the next fix). The large gains over "vLLM only" come from persistence itself, which both engines already ship; Lifeboat's additions (capacity limit, free-disk reserve, integrity, indexed manifest, ONTAP/NFS/S3 fallback, SGLang, dashboard) are operational.

Storage on that host: Lifeboat restore 4.8–5.2 GB/s from NVMe (raw device 6.7 GB/s), 17–19 GB/s from RAM (above a 100 Gb/s RDMA link), GPU H2D 28.7 GB/s; recovery of 100K blocks from the SQLite manifest 0.07 s vs 4.8 s by directory scan.

Qwen3-30B-A3B rows use LMCache mode; the native rows restored 95/95 blocks (4.9 GB) with a 95.5% external prefix-cache hit rate; greedy outputs matched the pre-restart outputs in 3/4 prompts (the 4th diverged mid-sentence — vLLM with fp8 KV is not bit-deterministic even without a cache).

• 99.7% of the prompt's KV served from Lifeboat after restart on the standard-attention model.
• Cold-request overhead ≈1% (LMCache mode); native mode +0.1–2% at 32K–128K, +9% at 8K (1.37 s vs 1.25 s); GPU-warm repeats unchanged.
• Restore throughput 12.2 GiB/s (zero-copy mmap, parallel BLAKE3, one copy); index rebuild for 16 GB of KV in 0.06 s.
• Hybrid (Mamba / linear-attention) models: on SGLang Lifeboat persists the attention KV and the recurrent state (HiCache v2 sidecar pools, state stored losslessly) → 1.8–5.8× after restart. On vLLM, native mode stores both through vLLM's own offloading → 5.3–32× after restart. Under LMCache they restore from 0.1.4 on (0.1.3 failed every hybrid restore), but LMCache's 20 GB L1 cannot stage a 256K hybrid prompt — prefer native mode for hybrid models.
• After a long prompt, allow SGLang's write-through (~30 s for 128K tokens) to finish before a restart; an unfinished tail is recomputed after the restart, never corrupted.

Compression — what the numbers say

The codecs apply to 16-bit KV. With an FP8 KV cache (--kv-cache-dtype fp8) blocks are stored as-is. fp8 is the default for every stored rung. The dashboard shows the sampled reconstruction error of whatever codec is live.

LBQ (0.1.5) is Lifeboat's low-bit codec. For each stored chunk it:

• centers the keys per chunk (attention ignores a shared key offset);
• applies a randomized Hadamard rotation;
• stores a per-vector fp16 scale and codes values with a Lloyd-Max codebook;
• uses separate key / value bit widths;
• keeps the first / last layer and the chunk that opens a sequence (the attention sink) at fp8.

Qwen3-4B, BF16 KV. Code and prose prompts are stored and restored through the live path in 256-token chunks, then a 512-token continuation is scored against the uncompressed cache (benchmarks/accuracy/codec_accuracy.py, docs/reports/codec_accuracy_qwen3-4b.json):

The old algorithms on Qwen3-30B-A3B (2K / 8K / 24K): turboquant4 2.7× at +0–4% perplexity; turboquant3 +44–64%; 2-bit OSCAR per sequence +955–1,379%. Per stored chunk, OSCAR compressed nothing (1.0×). LBQ has not yet been measured on the 30B model.

When to use which:

• LBQ decodes slower than fp8. Per 25 MB chunk (raw-equivalent): ~1 GB/s vs 4.3 GB/s for fp8 on eqprod's x86 CPU with 8 threads, and ~1.5 GB/s vs ~10 GB/s on an Apple M4 Max. Use it where bytes cost more than CPU: shared / archive rungs and capacity-bound disks.
• fp8 stays the default for local NVMe, where restore time matters most.
• storage_efficient uses lbq3 everywhere.