memkv-vllm

Two ways to back vLLM's native KV offloading with a shared MemKV cluster. Both ship in this one wheel and both are selected purely through --kv-transfer-config. Neither needs a patched vLLM to run; the tier's optional lookup_timeout_s deadline is the one exception, noted where it appears below.

They are alternatives — pick one per engine. The two modes write different key layouts, so a cluster written by one is not read by the other.

This is MemKV's third vLLM-adjacent integration and does not replace the others: memkv-lmcache (the LMCache connector path, currently the production recommendation) and the MemKV NIXL storage backend for Dynamo KVBM, which ships as a shared library rather than a wheel.

Requirements

• vLLM 0.25.0 or newer. Tested against 0.25.0, 0.25.1, 0.26.0, 0.27.0, and 0.27.1. Releases before 0.25.0 do not work: the tiering framework itself has shipped since 0.22.0, but the secondary-tier interfaces this plugin implements — LookupResult lookups, ScheduleEndContext, and get_stats/build_metric_definitions — arrived in 0.25.0. vLLM reworked the offloading-spec API again in 0.26.0; the plugin detects which side it is running against and adapts, so no version pin is needed in either direction. The offloading-spec API is still experimental upstream, so releases past 0.27.1 are untested.
• Python 3.10 or newer, on Linux (x86-64 or aarch64). The wheels on PyPI are manylinux_2_28, so glibc 2.28 or newer — RHEL/Rocky 8+, Debian 10+, Ubuntu 18.10+. (The copies bundled in the MemKV deb/rpm are built without a manylinux tag and target the same hosts.)
• RDMA userspace (libibverbs, ibverbs-providers) for the data path. The wheels load it lazily, so they import on any Linux host. What happens without a reachable NIC differs by mode: the secondary tier falls back to TCP, while direct offload is RDMA-only and fails at startup rather than running slowly.
• PYTHONHASHSEED pinned (e.g. 0) on every vLLM instance that should share KV — vLLM's block content hashes are seeded per process otherwise.
• --enable-cumem-allocator when PYTORCH_CUDA_ALLOC_CONF has expandable_segments:True. vLLM refuses to start OffloadingConnector under that setting otherwise. The flag puts the model weights and the KV cache in vLLM's own CuMem pool, where expandable segments are off; other allocations keep PYTORCH_CUDA_ALLOC_CONF as set.
• A MemKV license. The client verifies one when it builds its engine, so a plugin with none available fails at startup with no license found rather than degrading. Sources, in order: the license: field of MEMKV_CONFIG, then MEMKV_LICENSE (an inline JWT or a path to a file holding one), then MINIO_LICENSE / AISTOR_LICENSE and the standard minio.license file locations. Contact MinIO to obtain one.

Install

pip install memkv-vllm

The wheel registers itself via the vllm.general_plugins entry point memkv_tier — no vLLM patches, no extra flags. If you restrict plugins with VLLM_PLUGINS, include memkv_tier in the list.

Configure — secondary tier (TieringOffloadingSpec)

MemKV connection settings come from the standard MEMKV_CONFIG yaml or MEMKV_* env-var chain (MEMKV_SERVERS, MEMKV_AUTH_KEY, …) — identical to the other MemKV plugins. The tier entry in secondary_tiers carries only tier tuning:

PYTHONHASHSEED=0 vllm serve <model> \
  --kv-transfer-config '{
    "kv_connector": "OffloadingConnector",
    "kv_role": "kv_both",
    "kv_connector_extra_config": {
      "spec_name": "TieringOffloadingSpec",
      "cpu_bytes_to_use": 10737418240,
      "block_size": 256,
      "secondary_tiers": [
        {"type": "memkv", "n_read_threads": 8, "n_write_threads": 8}
      ]
    }
  }'

Sizing notes: block_size (offloaded block, in tokens) controls the MemKV value size — prefer values ≥ 1 MiB (long blocks) so reads are bandwidth-bound, not per-key-overhead-bound. cpu_bytes_to_use is the CPU tier working set; MemKV only sees traffic once blocks cascade (immediately on store) and promote (on CPU-tier misses).

Configure — direct offload (MemKVOffloadingSpec)

No CPU pool in front: the worker is handed the live GPU KV tensors, gathers each block into one scratch slot, and MemKV servers move it with one-sided RDMA. Connection settings come from the same MEMKV_* chain.

PYTHONHASHSEED=0 vllm serve <model> \
  --kv-transfer-config '{
    "kv_connector": "OffloadingConnector",
    "kv_role": "kv_both",
    "kv_connector_extra_config": {
      "spec_name": "MemKVOffloadingSpec",
      "spec_module_path": "memkv_vllm.spec"
    }
  }'

All keys sit directly in kv_connector_extra_config. Do not set block_size here — the offloaded block is the GPU block.

Each rank drives its RDMA through the NIC on its own GPU's PCIe switch, read from sysfs at startup; MEMKV_RDMA_DEVICES narrows the candidates if some NICs belong to another fabric. memkv net topo prints the layout.

memkv_scratch_medium follows from the same read. GPUDirect ("gpu") reaches line rate only when the NIC and that rank's GPU sit on the same PCIe switch; otherwise peer-to-peer climbs to the CPU root complex and back. The "auto" default takes "gpu" on the ranks that have such a NIC and "host" on the rest, so a host with more ranks than NICs needs no decision. Pin it to either value to override.

How it maps

Secondary tier. One MemKV value per (block hash, KV cache group): the full offloaded block across all TP ranks, stride bytes of the CPU pool. Keys reuse vLLM's FileMapper naming — <prefix>/<model>_<configdigest>…/<hash>.bin — so runs with the same model and layout share blocks, and incompatible layouts cannot collide.

Direct offload. One value per (block, rank), gathered from the live GPU KV tensors, so its length comes from the KV layout — the widest cache group's per-block size — not from a CPU-pool stride. Keys are binary and rank-qualified, under a digest of model, dtype, TP size, block size and layout; that is why a restore needs the same TP topology that stored it, and why a cluster written in this mode is not readable by the tier.

Both modes share the rest. Lookups are batched async exists probes, one round per scheduler step; stores and loads are chunked batch_put_views / batch_get_into calls with no Python-side copies. Capacity is governed server-side by MemKV lane eviction; a value evicted between lookup and load degrades to recompute, never corruption.

Test

cargo build -p memkv            # loopback server binary for the fixture
maturin develop --manifest-path vllm-plugin/Cargo.toml
pytest vllm-plugin/tests -v    # requires vllm importable; Linux for data path