Lifeboat Cache

Tiered, compressed, persistent KV cache for vLLM, SGLang, and Lifeboat (which embeds SGLang), with a built-in dashboard to configure, monitor and observe it.

pip install "lifeboatcache[vllm]"   # into the environment that runs vLLM
pip install lifeboatcache           # into the environment that runs SGLang / Lifeboat

Installing registers engine plugins; nothing else is needed to start. With no config file, Lifeboat writes KV to local disk under ~/.lifeboat_cache/kv, compresses it with fp8, and serves the dashboard on http://127.0.0.1:9400.

Turn it on

What happens underneath:

• vLLM — LifeboatConnector is LMCache's multiprocess connector with Lifeboat defaults: it auto-starts a local LMCache server (port 6555) whose L2 storage is the Lifeboat adapter. GPU ↔ host movement is LMCache's; everything below host RAM is Lifeboat.

• SGLang / Lifeboat — Lifeboat is a HiCache storage backend (--hicache-storage-backend dynamic). With LIFEBOAT_CACHE=1 the plugin enables HiCache and selects the backend unless the launch already chose a storage backend. Explicit flags work too:

  python -m sglang.launch_server --model-path <model> --enable-hierarchical-cache \
    --hicache-storage-backend dynamic \
    --hicache-storage-backend-extra-config '{"backend_name":"lifeboat","module_path":"lifeboat_cache.integrations.sglang_backend","class_name":"LifeboatHiCacheStorage"}'
  

Storage targets and the local fallback

Targets are tried in order; the first usable one takes new writes. Each is write-probed at startup. One that can't be reached (ONTAP/NFS export not mounted, S3 without credentials) is skipped with the reason shown on the dashboard, and if nothing persistent is left, KV goes to storage.fallback_local on local disk.

storage:
  tiers:
    - {name: ontap, type: ontap, enabled: true, mount: /mnt/kvcache, protocol: pnfs_rdma}
    - {name: nfs,   type: nfs,   enabled: true, mount: /mnt/nfs, path: /mnt/nfs/lifeboat/kv}
    - {name: s3,    type: s3,    enabled: true, bucket: kv-archive, region: us-east-1}
  fallback_local: {enabled: true, path: ~/.lifeboat_cache/kv, capacity_gb: 50}

Persisted KV survives restarts: on startup the index is rebuilt from disk in the background.

Dashboard

http://127.0.0.1:9400 on the node (use ssh -L 9400:localhost:9400 <node> from your laptop). It shows the cache hit rate, KV served, bytes saved by compression and the codec's sampled reconstruction error, per-target status (circuit breaker, usage, queue), evictions, and startup recovery. The Configure tab edits storage targets, the local fallback, the compression codec per rung, the eviction policy and the write mode; changes are validated, applied live, and saved to the config file (startup-bound sections are flagged "restart required"). /metrics serves Prometheus text (also on port 9402).

To expose it beyond localhost, set dashboard.host: 0.0.0.0 and dashboard.auth_token; open the page once with ?token=<token>.

CLI: lifeboat-cache status prints the same numbers; lifeboat-cache init writes a starter config; lifeboat-cache server runs a standalone Lifeboat-backed LMCache server for several vLLM instances to share.

Configuration

One YAML file, resolved as: --config / $LIFEBOAT_CACHE_CONFIG → ~/.lifeboat_cache/lifeboat.yaml → built-in defaults ($LIFEBOAT_CACHE_HOME moves the home directory). lifeboat-cache config show prints the resolved config. Presets ship with the package: extends: single_node_gpu.

Compression — what the numbers say

Measured on real Qwen3 KV (2,048-token prefix, see docs/):

fp8 is the default for every stored rung. The dashboard shows the sampled reconstruction error of whatever codec is live.