memkv-lmcache

LMCache StoragePluginInterface backend that persists KV chunks in a remote MemKV cluster. Loaded as a vendor plugin via LMCache's storage_plugins dynamic loader — no patches to LMCache's tree.

vLLM gets a MemKV-backed prefix KV-state path for free through vllm/distributed/kv_transfer/kv_connector/v1/lmcache_connector.py — no separate vLLM connector package is required.

Build

cd lmcache-plugin
pip install maturin
maturin develop --release      # local dev install
# or
maturin build --release        # wheel under target/wheels/
pip install target/wheels/memkv_lmcache-*.whl

The wheel bundles a native PyO3 extension built from the same memkv-client crate the NIXL plugin and the sglang plugin use, so RDMA/TCP transport selection works the same way across all three.

Configure the MemKV connection

The plugin reads the standard MemKV config chain — MEMKV_CONFIG yaml first, then MEMKV_* env vars:

export MEMKV_SERVERS="10.0.0.10:9900,10.0.0.11:9900"
export MEMKV_RDMA_DEVICES="mlx5_0,mlx5_1"
export MEMKV_AUTH_KEY="<64-hex>"
export MEMKV_LICENSE="/etc/memkv/minio.license"   # one way to supply the license
# optional:
# export MEMKV_TRANSPORT=auto
# export MEMKV_CONFIG=/etc/memkv.yaml

A license is required: the client verifies one when it builds its engine, and a plugin with no license available fails at startup with no license found rather than degrading. It is looked up in this order: the license: field of MEMKV_CONFIG, then MEMKV_LICENSE (an inline JWT or a path to a file holding one), then MINIO_LICENSE / AISTOR_LICENSE and the standard minio.license file locations. Contact MinIO to obtain one.

Configure LMCache

Add the plugin to your LMCache yaml. max_local_cpu_size must be > 0 because the plugin uses LocalCPUBackend's allocator to stage retrieved tensors:

chunk_size: 64
local_cpu: true
max_local_cpu_size: 5
storage_plugins: memkv
extra_config:
  storage_plugin.memkv.module_path: memkv_lmcache.backend
  storage_plugin.memkv.class_name: MemKVStorageBackend

Launch vLLM with LMCache + MemKV

LMCACHE_CONFIG_FILE=/etc/lmcache.yaml \
KV_TRANSFER_CONFIG='{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}' \
vllm serve meta-llama/Llama-3-8B \
    --tensor-parallel-size 1 \
    --kv-transfer-config "$KV_TRANSFER_CONFIG"

LMCache's connector picks the storage_plugins entry up at startup and routes prefix KV reads/writes through MemKVStorageBackend.

MP mode: MemKV as an L2 adapter

LMCache's multi-process mode (lmcache server + LMCacheMPConnector) — currently the only LMCache configuration that attaches to hybrid Mamba/attention models — persists L1 overflow through a different surface, the L2AdapterInterface. memkv_lmcache.l2_adapter implements it, loaded via LMCache's built-in plugin adapter type:

lmcache server --chunk-size <unified_block> --l1-size-gb 256 \
    --eviction-policy LRU \
    --l2-adapter '{"type":"plugin",
                   "module_path":"memkv_lmcache.l2_adapter",
                   "class_name":"MemKVL2Adapter",
                   "adapter_params":{"prefix":"fleet-a"}}'

The MemKV connection comes from the same MEMKV_CONFIG / MEMKV_* env chain; adapter_params carries only adapter tuning (prefix, num_workers, blocks_per_op, load_failure_grace_s — see the module docstring). Unlike the classic backend, the L2 port needs no shape sidecars (the controller owns every buffer), so values written by one server instance are readable by any other — including across lmcache server restarts.

What's implemented

Caveats

• Cross-restart warm cache is limited. Each engine process keeps shape/dtype/fmt in an in-memory dict so get_blocking knows what MemoryObj to allocate. The wire bytes survive in MemKV across restarts; the local metadata does not. A fresh process therefore starts cold even when MemKV holds the chunks. This matches LMCache's LocalDiskBackend behavior. Encoding the shape header on the wire is a follow-up.
• Key length cap. MemKV's protocol caps keys at 512 bytes, so a CacheEngineKey.to_string() value longer than 480 bytes collapses to a memkv-h2:<blake2b-256> digest.
• Pin/unpin are local-only. MemKV has no per-client retention policy, so the methods are presence checks against the local meta dict. Server-side eviction is owned by the MemKV cluster.
• Reads ride the server-driven chunked BatchRead. get_blocking uses batch_get_into, which streams the full value through the server's bounce buffers into per-thread staging with strict full-length-or-miss semantics. The single-key zero-copy get_into (client-driven RDMA READ) remains available but is not the LMCache default: under sustained burst load it saturated the per-connection RC send CQ tail and tripped vLLM's SPMD broadcast timeout, and it faults the value resident server-side.

Layout

lmcache-plugin/
├── Cargo.toml                      # cdylib + pyo3 + memkv-client
├── pyproject.toml                  # maturin
├── src/lib.rs                      # PyO3 wrapper around memkv-client::Engine
└── python/memkv_lmcache/
    ├── __init__.py                 # re-exports Client
    └── backend.py                  # MemKVStorageBackend(StoragePluginInterface)