memkv-vllm

MemKV secondary tier for vLLM's native KV offloading feature (OffloadingConnector + TieringOffloadingSpec). vLLM offloads completed KV blocks from GPU to a pinned CPU pool; this plugin adds a memkv tier behind that pool, so blocks evicted from CPU (or produced by another vLLM instance) are served from a shared MemKV cluster over RDMA instead of being recomputed.

This is MemKV's third vLLM-adjacent integration and does not replace the others: lmcache-plugin/ (LMCache connector path, current production recommendation) and plugin/ (NIXL storage backend for Dynamo KVBM).

Requirements

• vLLM main at or after commit 6cf7b26bd (v0.23.1rc1.dev962). The tiering framework (vllm.v1.kv_offload.tiering) has shipped in releases since v0.22.0, but the secondary-tier API this plugin implements (LookupResult lookups, ScheduleEndContext, get_stats/build_metric_definitions) is newer than the latest stable release (v0.24.0); it first appears in the v0.25.0 release candidates. The offloading spec API is experimental upstream and has already been reworked once (a9531edfa, between v0.25.1 and v0.26.0 — renamed OffloadingSpec construction and several fields this plugin reads). Rather than pinning to one side of that boundary, _vllm_compat.py detects and handles both; confirmed working on v0.25.0, v0.25.1, v0.26.0, v0.27.0, and v0.27.1.
• Linux with RDMA userspace (libibverbs, ibverbs-providers) for the data path. The client falls back to TCP where RDMA is unavailable.
• PYTHONHASHSEED pinned (e.g. 0) on every vLLM instance that should share KV — vLLM's block content hashes are seeded per process otherwise.

Install

pip install memkv-vllm

The wheel registers itself via the vllm.general_plugins entry point memkv_tier — no vLLM patches, no extra flags. If you restrict plugins with VLLM_PLUGINS, include memkv_tier in the list.

Configure

MemKV connection settings come from the standard MEMKV_CONFIG yaml or MEMKV_* env-var chain (MEMKV_SERVERS, MEMKV_AUTH_KEY, …) — identical to the other MemKV plugins. The tier entry in secondary_tiers carries only tier tuning:

PYTHONHASHSEED=0 vllm serve <model> \
  --kv-transfer-config '{
    "kv_connector": "OffloadingConnector",
    "kv_role": "kv_both",
    "kv_connector_extra_config": {
      "spec_name": "TieringOffloadingSpec",
      "cpu_bytes_to_use": 10737418240,
      "block_size": 256,
      "secondary_tiers": [
        {"type": "memkv", "n_read_threads": 8, "n_write_threads": 8}
      ]
    }
  }'

Sizing notes: block_size (offloaded block, in tokens) controls the MemKV value size — prefer values ≥ 1 MiB (long blocks) so reads are bandwidth-bound, not per-key-overhead-bound. cpu_bytes_to_use is the CPU tier working set; MemKV only sees traffic once blocks cascade (immediately on store) and promote (on CPU-tier misses).

How it maps

One MemKV value per (block hash, KV cache group): the full offloaded block across all TP ranks (stride bytes of the CPU pool). Keys reuse vLLM's FileMapper naming — <prefix>/<model>_<configdigest>…/<hash>.bin — so runs with the same model/layout share blocks and incompatible layouts can't collide. Lookups are batched async exists probes (one round per scheduler step); stores/loads are chunked batch_put_views/batch_get_into calls straight against the pool slots (no Python-side copies). Capacity is governed server-side by MemKV lane eviction; a value evicted between lookup and load degrades to recompute, never corruption.

Test

cargo build -p memkv            # loopback server binary for the fixture
maturin develop --manifest-path vllm-plugin/Cargo.toml
pytest vllm-plugin/tests -v    # requires vllm importable; Linux for data path