Skip to main content

Dynamo KVBM

The Dynamo KVBM is a distributed KV-cache block management system designed for scalable LLM inference. It cleanly separates memory management from inference runtimes (vLLM, TensorRT-LLM, and SGLang), enabling GPU↔CPU↔Disk/Remote tiering, asynchronous block offload/onboard, and efficient block reuse.

A block diagram showing a layered architecture view of Dynamo KV Block manager.

Feature Highlights

  • Distributed KV-Cache Management: Unified GPU↔CPU↔Disk↔Remote tiering for scalable LLM inference.
  • Async Offload & Reuse: Seamlessly move KV blocks between memory tiers using GDS-accelerated transfers powered by NIXL, without recomputation.
  • Runtime-Agnostic: Works out-of-the-box with vLLM, TensorRT-LLM, and SGLang via lightweight connectors.
  • Memory-Safe & Modular: RAII lifecycle and pluggable design for reliability, portability, and backend extensibility.

Build and Installation

The pip wheel is built through a Docker build process:

# Build the Docker image with KVBM enabled (from the dynamo repo root)
./container/build.sh --framework none --enable-kvbm --tag local-kvbm

Once built, you can either:

Option 1: Run and use the container directly

./container/run.sh --framework none -it

Option 2: Extract the wheel file to your local filesystem

# Create a temporary container from the built image
docker create --name temp-kvbm-container local-kvbm:latest

# Copy the KVBM wheel to your current directory
docker cp temp-kvbm-container:/opt/dynamo/wheelhouse/ ./dynamo_wheelhouse

# Clean up the temporary container
docker rm temp-kvbm-container

# Install the wheel locally
pip install ./kvbm*.whl

Note that the default pip wheel built is not compatible with CUDA 13 at the moment.

Integrations

Environment Variables

Variable Description Default
DYN_KVBM_CPU_CACHE_GB CPU pinned memory cache size (GB) required
DYN_KVBM_DISK_CACHE_GB SSD Disk/Storage system cache size (GB) optional
DYN_KVBM_LEADER_WORKER_INIT_TIMEOUT_SECS Timeout (in seconds) for the KVBM leader and worker to synchronize and allocate the required memory and storage. Increase this value if allocating large amounts of memory or storage. 120
DYN_KVBM_METRICS Enable metrics endpoint false
DYN_KVBM_METRICS_PORT Metrics port 6880
DYN_KVBM_DISABLE_DISK_OFFLOAD_FILTER Disable disk offload filtering to remove SSD lifespan protection false

vLLM

DYN_KVBM_CPU_CACHE_GB=100 vllm serve \
  --kv-transfer-config '{"kv_connector":"DynamoConnector","kv_role":"kv_both","kv_connector_module_path":"kvbm.vllm_integration.connector"}' \
  Qwen/Qwen3-8B

For more detailed integration with dynamo, disaggregated serving support and benchmarking, please check vllm-setup

TensorRT-LLM

cat >/tmp/kvbm_llm_api_config.yaml <<EOF
cuda_graph_config: null
kv_cache_config:
  enable_partial_reuse: false
  free_gpu_memory_fraction: 0.80
kv_connector_config:
  connector_module: kvbm.trtllm_integration.connector
  connector_scheduler_class: DynamoKVBMConnectorLeader
  connector_worker_class: DynamoKVBMConnectorWorker
EOF

DYN_KVBM_CPU_CACHE_GB=100 trtllm-serve Qwen/Qwen3-8B \
  --host localhost --port 8000 \
  --backend pytorch \
  --extra_llm_api_options /tmp/kvbm_llm_api_config.yaml

For more detailed integration with dynamo and benchmarking, please check trtllm-setup

📚 Docs

Metadata

Release files for kvbm 0.7.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for kvbm 0.7.0
File Interpreter ABI Platform Actions
kvbm-0.7.0-cp310-abi3-manylinux_2_28_x86_64.whl CPython 3.10 abi3 Linux glibc 2.28+ x86-64 Details for kvbm-0.7.0-cp310-abi3-manylinux_2_28_x86_64.whl
kvbm-0.7.0-cp310-abi3-manylinux_2_28_aarch64.whl CPython 3.10 abi3 Linux glibc 2.28+ ARM64 Details for kvbm-0.7.0-cp310-abi3-manylinux_2_28_aarch64.whl

Total release size: 16.2 MB

Release files / kvbm-0.7.0-cp310-abi3-manylinux_2_28_x86_64.whl

Download URL kvbm-0.7.0-cp310-abi3-manylinux_2_28_x86_64.whl
Size 8.2 MB
Tags CPython 3.10 Linux glibc 2.28+ x86-64 abi3
SHA-256 checksum
How to use checksums
34af9f6a2008cbe5ae754273060d11bc10b5e8a44e8745310c45ccd4b4dc3bad
BLAKE2b-256 checksum
How to use checksums
e0dae0bbebe9c7dd085f7320b2de36d3e67df7ca442ac1c3b739a173ded3cef3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.0

Release files / kvbm-0.7.0-cp310-abi3-manylinux_2_28_aarch64.whl

Download URL kvbm-0.7.0-cp310-abi3-manylinux_2_28_aarch64.whl
Size 8.0 MB
Tags CPython 3.10 Linux glibc 2.28+ ARM64 abi3
SHA-256 checksum
How to use checksums
51f8e16b48133d0f600740b9c1e9ec50197e72417783e2366cb5aed175d01ad8
BLAKE2b-256 checksum
How to use checksums
8740fbc44c59718b20c523c2b5449fd310bc1b40a9b8fdbb55b865e1f0e926c4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.0

Security for kvbm 0.7.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.7.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page