Lifeboat Cache

Tiered, compressed, persistent KV cache for vLLM, SGLang, and Lifeboat (which embeds SGLang), with a built-in dashboard to configure, monitor and observe it.

pip install "lifeboatcache[vllm]"   # into the environment that runs vLLM
pip install lifeboatcache           # into the environment that runs SGLang / Lifeboat

Installing registers engine plugins; nothing else is needed to start. With no config file, Lifeboat writes KV to local disk under ~/.lifeboat_cache/kv, compresses it with fp8, and serves the dashboard on http://127.0.0.1:9400.

Turn it on

What happens underneath:

• vLLM (default, engine.vllm_mode: native) — Lifeboat is a secondary tier of vLLM's own KV offloading (OffloadingConnector + TieringOffloadingSpec). vLLM moves GPU ↔ CPU for every KV cache group, including hybrid models' Mamba / linear-attention state; Lifeboat persists those blocks. No LMCache needed.

• vLLM (engine.vllm_mode: lmcache) — LifeboatConnector is LMCache's multiprocess connector with Lifeboat defaults: it auto-starts a local LMCache server (port 6555) whose L2 storage is the Lifeboat adapter. Standard-attention models only.

• SGLang / Lifeboat — Lifeboat is a HiCache storage backend (--hicache-storage-backend dynamic). With LIFEBOAT_CACHE=1 the plugin enables HiCache and selects the backend unless the launch already chose a storage backend. Explicit flags work too:

  python -m sglang.launch_server --model-path <model> --enable-hierarchical-cache \
    --hicache-storage-backend dynamic \
    --hicache-storage-backend-extra-config '{"backend_name":"lifeboat","module_path":"lifeboat_cache.integrations.sglang_backend","class_name":"LifeboatHiCacheStorage"}'
  

vLLM: install and start

# into the Python environment that runs vLLM (vLLM >= 0.32; LMCache not required)
pip install lifeboatcache

# zero-flag: the plugin wires vLLM's native offloading -> CPU tier -> Lifeboat Cache
LIFEBOAT_CACHE=1 vllm serve /models/Qwen3.8-27B --served-model-name my-model \
  --kv-cache-dtype fp8 --enable-prefix-caching

# explicit (same result, no env var)
vllm serve /models/Qwen3.8-27B --served-model-name my-model --enable-prefix-caching \
  --kv-transfer-config '{"kv_connector":"OffloadingConnector","kv_role":"kv_both",
    "kv_connector_extra_config":{"spec_name":"TieringOffloadingSpec","cpu_bytes_to_use":21474836480,
      "secondary_tiers":[{"type":"LifeboatTier","module_path":"lifeboat_cache.integrations.vllm_tier"}]}}'

How it attaches, native mode (no vLLM source changes):

vLLM (GPU KV + Mamba state, prefix cache)
  └─ OffloadingConnector / TieringOffloadingSpec                       [vllm.general_plugins]
       └─ CPU primary tier (/dev/shm, engine.native_cpu_gb, default 20 GB)
            └─ secondary tier: LifeboatTier                 [module_path plugin]
                 └─ tiers: ONTAP / NFS / S3 / local disk (fallback), index, eviction,
                    dashboard :9400, Prometheus :9402

The CPU tier lives in /dev/shm: make sure it has native_cpu_gb free (a vLLM killed with SIGKILL can leave a stale /dev/shm/vllm_offload_*.mmap behind — delete it if nothing maps it). Blocks are stored raw (KV is already fp8 with --kv-cache-dtype fp8; recurrent state must stay lossless), keyed by vLLM's storage namespace (model, parallel layout, dtype, KV groups) so they are never served to an incompatible configuration.

LMCache mode (engine.vllm_mode: lmcache, pip install "lifeboatcache[vllm]", or --kv-transfer-config '{"kv_connector":"LifeboatConnector","kv_role":"kv_both"}'):

vLLM (GPU KV, prefix cache)
  └─ LifeboatConnector  = LMCache MP connector + Lifeboat defaults      [vllm.general_plugins]
       └─ LMCache MP server (auto-started, localhost:6555)  L1 = host RAM
            └─ L2 adapter: LifeboatL2Adapter                 [--l2-adapter plugin]
                 └─ tiers: ONTAP / NFS / S3 / local disk (fallback), compression, index,
                    eviction policy, dashboard :9400, Prometheus :9402

Hybrid (Mamba / linear-attention) models such as Qwen3.5/3.8: use native mode (the default) — 5.3–16.6× faster first token after a restart on Qwen3.8-27B, see below. Keep --max-num-seqs below the Mamba cache-block count. Behind LMCache 0.5.6 these models reuse nothing (external hit rate 0%) because LMCache does not pass the recurrent state down.

If vLLM already uses LMCache

Lifeboat Cache never replaces a KV connector you configured — an explicit --kv-transfer-config always wins, LIFEBOAT_CACHE=1 included.

Storage targets and the local fallback

Targets are tried in order; the first usable one takes new writes. Each is write-probed at startup. One that can't be reached (ONTAP/NFS export not mounted, S3 without credentials) is skipped with the reason shown on the dashboard, and if nothing persistent is left, KV goes to storage.fallback_local on local disk.

storage:
  tiers:
    - {name: ontap, type: ontap, enabled: true, mount: /mnt/kvcache, protocol: pnfs_rdma}
    - {name: nfs,   type: nfs,   enabled: true, mount: /mnt/nfs, path: /mnt/nfs/lifeboat/kv}
    - {name: s3,    type: s3,    enabled: true, bucket: kv-archive, region: us-east-1}
  fallback_local: {enabled: true, path: ~/.lifeboat_cache/kv, capacity_gb: 50}

Persisted KV survives restarts: on startup the index is rebuilt from disk in the background.

Dashboard

http://127.0.0.1:9400 on the node (use ssh -L 9400:localhost:9400 <node> from your laptop). It shows the cache hit rate, KV served, bytes saved by compression and the codec's sampled reconstruction error, per-target status (circuit breaker, usage, queue), evictions, and startup recovery. The Configure tab edits storage targets, the local fallback, the compression codec per rung, the eviction policy and the write mode; changes are validated, applied live, and saved to the config file (startup-bound sections are flagged "restart required"). /metrics serves Prometheus text (also on port 9402).

To expose it beyond localhost, set dashboard.host: 0.0.0.0 and dashboard.auth_token; open the page once with ?token=<token>.

CLI: lifeboat-cache status prints the same numbers; lifeboat-cache init writes a starter config; lifeboat-cache server runs a standalone Lifeboat-backed LMCache server for several vLLM instances to share.

Configuration

One YAML file, resolved as: --config / $LIFEBOAT_CACHE_CONFIG → ~/.lifeboat_cache/lifeboat.yaml → built-in defaults ($LIFEBOAT_CACHE_HOME moves the home directory). lifeboat-cache config show prints the resolved config. Presets ship with the package: extends: single_node_gpu.

Measured results (vLLM 0.32 / SGLang 0.5.19, 1× RTX PRO 6000, FP8 KV)

Time to first token for the same prompt after a full server restart (real source-code prompts):

Qwen3-30B-A3B rows use LMCache mode; the native rows restored 95/95 blocks (4.9 GB) with a 95.5% external prefix-cache hit rate; greedy outputs matched the pre-restart outputs in 3/4 prompts (the 4th diverged mid-sentence — vLLM with fp8 KV is not bit-deterministic even without a cache).

• 99.7% of the prompt's KV served from Lifeboat after restart on the standard-attention model.
• Cold-request overhead ≈1% (LMCache mode); native mode +0.1–2% at 32K–128K, +9% at 8K (1.37 s vs 1.25 s); GPU-warm repeats unchanged.
• Restore throughput 12.2 GiB/s (zero-copy mmap, parallel BLAKE3, one copy); index rebuild for 16 GB of KV in 0.06 s.
• Hybrid (Mamba / linear-attention) models: on SGLang Lifeboat persists the attention KV and the recurrent state (HiCache v2 sidecar pools, state stored losslessly) → 1.8–5.8× after restart. On vLLM, native mode stores both through vLLM's own offloading → 5.3–16.6× after restart. Behind LMCache 0.5.6 they reuse nothing (no recurrent state) — don't use vllm_mode: lmcache for hybrid models.
• After a long prompt, allow SGLang's write-through (~30 s for 128K tokens) to finish before a restart; an unfinished tail is recomputed after the restart, never corrupted.

Compression — what the numbers say

Measured on real Qwen3 KV (2,048-token prefix, see docs/):

fp8 is the default for every stored rung. The dashboard shows the sampled reconstruction error of whatever codec is live.