Dynamo KVBM

The Dynamo KVBM is a distributed KV-cache block management system designed for scalable LLM inference. It cleanly separates memory management from inference runtimes (vLLM, TensorRT-LLM, and SGLang), enabling GPU↔CPU↔Disk/Remote tiering, asynchronous block offload/onboard, and efficient block reuse.

A block diagram showing a layered architecture view of Dynamo KV Block manager.

Feature Highlights

Distributed KV-Cache Management: Unified GPU↔CPU↔Disk↔Remote tiering for scalable LLM inference.
Async Offload & Reuse: Seamlessly move KV blocks between memory tiers using GDS-accelerated transfers powered by NIXL, without recomputation.
Runtime-Agnostic: Works out-of-the-box with vLLM, TensorRT-LLM, and SGLang via lightweight connectors.
Memory-Safe & Modular: RAII lifecycle and pluggable design for reliability, portability, and backend extensibility.

Installation

pip install kvbm

See the support matrix for version compatibility questions.

Build from Source

The pip wheel is built through a Docker build process:

# Render and build the Docker image with KVBM enabled (from the dynamo repo root)
python container/render.py --framework dynamo --target runtime --output-short-filename
docker build --build-arg ENABLE_KVBM="true" -f container/rendered.Dockerfile .

Once built, you can either:

Option 1: Run and use the container directly

./container/run.sh --framework none -it

Option 2: Extract the wheel file to your local filesystem

# Create a temporary container from the built image
docker create --name temp-kvbm-container local-kvbm:latest

# Copy the KVBM wheel to your current directory
docker cp temp-kvbm-container:/opt/dynamo/wheelhouse/ ./dynamo_wheelhouse

# Clean up the temporary container
docker rm temp-kvbm-container

# Install the wheel locally
pip install ./dynamo_wheelhouse/kvbm*.whl

Note that the default pip wheel built is not compatible with CUDA 13 at the moment.

Integrations

Environment Variables

Variable	Description	Default
`DYN_KVBM_CPU_CACHE_GB`	CPU pinned memory cache size (GB)	required
`DYN_KVBM_DISK_CACHE_GB`	SSD Disk/Storage system cache size (GB)	optional
`DYN_KVBM_DISK_CACHE_DIR`	Disk cache directory	`/tmp/`
`DYN_KVBM_DISK_ZEROFILL_FALLBACK`	Enable zero-fill when `fallocate()` unsupported (e.g., Lustre)	`false`
`DYN_KVBM_DISK_DISABLE_O_DIRECT`	Disable O_DIRECT for disk I/O (debug/compatibility)	`false`
`DYN_KVBM_LEADER_WORKER_INIT_TIMEOUT_SECS`	Timeout (in seconds) for the KVBM leader and worker to synchronize and allocate the required memory and storage. Increase this value if allocating large amounts of memory or storage.	120
`DYN_KVBM_METRICS`	Enable metrics endpoint	`false`
`DYN_KVBM_METRICS_PORT`	Metrics port	`6880`
`DYN_KVBM_DISABLE_DISK_OFFLOAD_FILTER`	Disable disk offload filtering to remove SSD lifespan protection	`false`
`DYN_KVBM_HOST_OFFLOAD_PREFIX_MIN_PRIORITY`	Minimum priority (0-100) for CPU offload with contiguous (prefix) semantics: offloading stops at the first block below threshold, and all subsequent blocks are also skipped. Used for priority-based filtering.	`0` (no filtering)
`DYN_KVBM_NCCL_MLA_MODE`	Enable NCCL replicated mode for MLA (Multi-Layer Attention) models (e.g., DeepSeek). When set to `true`, rank 0 loads KV blocks from G2/G3 storage and broadcasts to all GPUs via NCCL instead of each GPU loading independently. Requires MPI and optional `nccl` feature for optimal behavior.	`false`

Disk Storage Configuration

Why special configuration may be needed:

Some filesystems (e.g., Lustre, certain network filesystems) don't support fallocate(), which KVBM uses for fast disk space allocation. Additionally, KVBM uses O_DIRECT I/O for GPU DirectStorage (GDS) performance, which requires strict 4096-byte alignment.

Setup for filesystems without fallocate() support:

export DYN_KVBM_DISK_CACHE_DIR=/mnt/storage/kvbm_cache
export DYN_KVBM_DISK_ZEROFILL_FALLBACK=true  # Enables zero-fill fallback when fallocate() unsupported

What happens:

Without ZEROFILL_FALLBACK=true: Disk cache allocation may fail with "Operation not supported"
With ZEROFILL_FALLBACK=true: KVBM writes zeros using page-aligned buffers compatible with O_DIRECT requirements

Troubleshooting: If you encounter "write all error" or EINVAL (errno 22), try disabling O_DIRECT: export DYN_KVBM_DISK_DISABLE_O_DIRECT=true

vLLM

DYN_KVBM_CPU_CACHE_GB=100 vllm serve \
  --kv-transfer-config '{"kv_connector":"DynamoConnector","kv_role":"kv_both","kv_connector_module_path":"kvbm.vllm_integration.connector"}' \
  Qwen/Qwen3-8B

See the KVBM configuration reference for all KVBM settings.

TensorRT-LLM

cat >/tmp/kvbm_llm_api_config.yaml <<EOF
cuda_graph_config: null
kv_cache_config:
  enable_partial_reuse: false
  free_gpu_memory_fraction: 0.80
kv_connector_config:
  connector_module: kvbm.trtllm_integration.connector
  connector_scheduler_class: DynamoKVBMConnectorLeader
  connector_worker_class: DynamoKVBMConnectorWorker
EOF

DYN_KVBM_CPU_CACHE_GB=100 trtllm-serve Qwen/Qwen3-8B \
  --host localhost --port 8000 \
  --backend pytorch \
  --extra_llm_api_options /tmp/kvbm_llm_api_config.yaml

See the KVBM configuration reference for all KVBM settings.

📚 Docs

Release files for kvbm 1.5.0.dev20260903

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for kvbm 1.5.0.dev20260903
File	Interpreter	ABI	Platform	Reset
kvbm-1.5.0.dev20260903-cp310-abi3-manylinux_2_28_x86_64.whl 12.6 MB Sep 3, 2026	CPython 3.10	abi3	Linux glibc 2.28+ x86-64	Details
kvbm-1.5.0.dev20260903-cp310-abi3-manylinux_2_28_aarch64.whl 10.2 MB Sep 3, 2026	CPython 3.10	abi3	Linux glibc 2.28+ ARM64	Details

Total release size: 22.8 MB

Release files / kvbm-1.5.0.dev20260903-cp310-abi3-manylinux_2_28_x86_64.whl

Download URL	kvbm-1.5.0.dev20260903-cp310-abi3-manylinux_2_28_x86_64.whl
Size	12.6 MB
Tags	CPython 3.10 Linux glibc 2.28+ x86-64 abi3
SHA-256 checksum How to use checksums	`78b766f84e6538dd42522a2b20e2c470d260c067b88c4cff04e90b11cf834aef`
BLAKE2b-256 checksum How to use checksums	`2484ec95a925aea125eae3b82b175f955386d29b738b88feebc18b62fb281db1`
Upload date	Sep 3, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.14.7`

Release files / kvbm-1.5.0.dev20260903-cp310-abi3-manylinux_2_28_aarch64.whl

Download URL	kvbm-1.5.0.dev20260903-cp310-abi3-manylinux_2_28_aarch64.whl
Size	10.2 MB
Tags	CPython 3.10 Linux glibc 2.28+ ARM64 abi3
SHA-256 checksum How to use checksums	`8ff39add8f789405dcc2c392ec2298d81be601b0001276d10e465b331941acd4`
BLAKE2b-256 checksum How to use checksums	`508af6de61f73ea6c97385f5bed7f70211d590a343c1996c725dd10dac9aa7b2`
Upload date	Sep 3, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.14.7`