memkv-vllm 1.0.4

vLLM native KV offloading secondary tier backed by the MemKV context memory store (TieringOffloadingSpec).

memkv-vllm

MemKV secondary tier for vLLM's native KV offloading feature (OffloadingConnector + TieringOffloadingSpec). vLLM offloads completed KV blocks from GPU to a pinned CPU pool; this plugin adds a memkv tier behind that pool, so blocks evicted from CPU (or produced by another vLLM instance) are served from a shared MemKV cluster over RDMA instead of being recomputed.

This is MemKV's third vLLM-adjacent integration and does not replace the others: lmcache-plugin/ (LMCache connector path, current production recommendation) and plugin/ (NIXL storage backend for Dynamo KVBM).

Requirements

• vLLM main at or after commit 6cf7b26bd (v0.23.1rc1.dev962). The tiering framework (vllm.v1.kv_offload.tiering) has shipped in releases since v0.22.0, but the secondary-tier API this plugin implements (LookupResult lookups, ScheduleEndContext, get_stats/build_metric_definitions) is newer than the latest stable release (v0.24.0); it first appears in the v0.25.0 release candidates. The offloading spec API is experimental upstream; this plugin pins to the commit range noted in plans/VLLM-TIERING-PLUGIN-PLAN.md.
• Linux with RDMA userspace (libibverbs, ibverbs-providers) for the data path. The client falls back to TCP where RDMA is unavailable.
• PYTHONHASHSEED pinned (e.g. 0) on every vLLM instance that should share KV — vLLM's block content hashes are seeded per process otherwise.

Install

pip install memkv-vllm

The wheel registers itself via the vllm.general_plugins entry point memkv_tier — no vLLM patches, no extra flags. If you restrict plugins with VLLM_PLUGINS, include memkv_tier in the list.

Configure

MemKV connection settings come from the standard MEMKV_CONFIG yaml or MEMKV_* env-var chain (MEMKV_SERVERS, MEMKV_AUTH_KEY, …) — identical to the other MemKV plugins. The tier entry in secondary_tiers carries only tier tuning:

PYTHONHASHSEED=0 vllm serve <model> \
  --kv-transfer-config '{
    "kv_connector": "OffloadingConnector",
    "kv_role": "kv_both",
    "kv_connector_extra_config": {
      "spec_name": "TieringOffloadingSpec",
      "cpu_bytes_to_use": 10737418240,
      "block_size": 256,
      "secondary_tiers": [
        {"type": "memkv", "n_read_threads": 8, "n_write_threads": 8}
      ]
    }
  }'

Tier config key	Default	Meaning
type	—	Must be "memkv".
prefix	""	Key-namespace prefix inside MemKV (multi-fleet separation).
n_read_threads	8	Load-priority I/O threads (promotions, TTFT-critical).
n_write_threads	8	Store-priority I/O threads (cascades).
blocks_per_op	8	Max blocks per batched client call.
load_failure_grace_s	60	Withhold a failed load's completion from vLLM for this long, keeping its target slots pinned so a server write abandoned by a client timeout cannot land in reused memory. 0 disables.
max_store_backlog_mb	2048	Shed store jobs once this much store work is queued/in flight, so a slow MemKV cannot pin the CPU pool solid. Shed blocks just skip MemKV; loads are never shed. Keep below cpu_bytes_to_use/2.

Sizing notes: block_size (offloaded block, in tokens) controls the MemKV value size — prefer values ≥ 1 MiB (long blocks) so reads are bandwidth-bound, not per-key-overhead-bound. cpu_bytes_to_use is the CPU tier working set; MemKV only sees traffic once blocks cascade (immediately on store) and promote (on CPU-tier misses).

How it maps

One MemKV value per (block hash, KV cache group): the full offloaded block across all TP ranks (stride bytes of the CPU pool). Keys reuse vLLM's FileMapper naming — <prefix>/<model>_<configdigest>…/<hash>.bin — so runs with the same model/layout share blocks and incompatible layouts can't collide. Lookups are batched async exists probes (one round per scheduler step); stores/loads are chunked batch_put_views/batch_get_into calls straight against the pool slots (no Python-side copies). Capacity is governed server-side by MemKV lane eviction; a value evicted between lookup and load degrades to recompute, never corruption.

Test

cargo build -p memkv            # loopback server binary for the fixture
maturin develop --manifest-path vllm-plugin/Cargo.toml
pytest vllm-plugin/tests -v    # requires vllm importable; Linux for data path