Dynamo KVBM
The Dynamo KVBM is a distributed KV-cache block management system designed for scalable LLM inference. It cleanly separates memory management from inference runtimes (vLLM, TensorRT-LLM, and SGLang), enabling GPU↔CPU↔Disk/Remote tiering, asynchronous block offload/onboard, and efficient block reuse.
Feature Highlights
- Distributed KV-Cache Management: Unified GPU↔CPU↔Disk↔Remote tiering for scalable LLM inference.
- Async Offload & Reuse: Seamlessly move KV blocks between memory tiers using GDS-accelerated transfers powered by NIXL, without recomputation.
- Runtime-Agnostic: Works out-of-the-box with vLLM, TensorRT-LLM, and SGLang via lightweight connectors.
- Memory-Safe & Modular: RAII lifecycle and pluggable design for reliability, portability, and backend extensibility.
Build and Installation
The pip wheel is built through a Docker build process:
# Build the Docker image with KVBM enabled (from the dynamo repo root)
./container/build.sh --framework none --enable-kvbm --tag local-kvbm
Once built, you can either:
Option 1: Run and use the container directly
./container/run.sh --framework none -it
Option 2: Extract the wheel file to your local filesystem
# Create a temporary container from the built image
docker create --name temp-kvbm-container local-kvbm:latest
# Copy the KVBM wheel to your current directory
docker cp temp-kvbm-container:/opt/dynamo/wheelhouse/ ./dynamo_wheelhouse
# Clean up the temporary container
docker rm temp-kvbm-container
# Install the wheel locally
pip install ./kvbm*.whl
Note that the default pip wheel built is not compatible with CUDA 13 at the moment.
Integrations
Environment Variables
| Variable | Description | Default |
|---|---|---|
DYN_KVBM_CPU_CACHE_GB |
CPU pinned memory cache size (GB) | required |
DYN_KVBM_DISK_CACHE_GB |
SSD Disk/Storage system cache size (GB) | optional |
DYN_KVBM_LEADER_WORKER_INIT_TIMEOUT_SECS |
Timeout (in seconds) for the KVBM leader and worker to synchronize and allocate the required memory and storage. Increase this value if allocating large amounts of memory or storage. | 120 |
DYN_KVBM_METRICS |
Enable metrics endpoint | false |
DYN_KVBM_METRICS_PORT |
Metrics port | 6880 |
DYN_KVBM_DISABLE_DISK_OFFLOAD_FILTER |
Disable disk offload filtering to remove SSD lifespan protection | false |
vLLM
DYN_KVBM_CPU_CACHE_GB=100 vllm serve \
--kv-transfer-config '{"kv_connector":"DynamoConnector","kv_role":"kv_both","kv_connector_module_path":"kvbm.vllm_integration.connector"}' \
Qwen/Qwen3-8B
For more detailed integration with dynamo, disaggregated serving support and benchmarking, please check vllm-setup
TensorRT-LLM
cat >/tmp/kvbm_llm_api_config.yaml <<EOF
cuda_graph_config: null
kv_cache_config:
enable_partial_reuse: false
free_gpu_memory_fraction: 0.80
kv_connector_config:
connector_module: kvbm.trtllm_integration.connector
connector_scheduler_class: DynamoKVBMConnectorLeader
connector_worker_class: DynamoKVBMConnectorWorker
EOF
DYN_KVBM_CPU_CACHE_GB=100 trtllm-serve Qwen/Qwen3-8B \
--host localhost --port 8000 \
--backend pytorch \
--extra_llm_api_options /tmp/kvbm_llm_api_config.yaml
For more detailed integration with dynamo and benchmarking, please check trtllm-setup
📚 Docs
Metadata
Release files for kvbm 0.7.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distributions (wheels)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| kvbm-0.7.0-cp310-abi3-manylinux_2_28_x86_64.whl | CPython 3.10 | abi3 | Linux glibc 2.28+ x86-64 | Details for kvbm-0.7.0-cp310-abi3-manylinux_2_28_x86_64.whl |
| kvbm-0.7.0-cp310-abi3-manylinux_2_28_aarch64.whl | CPython 3.10 | abi3 | Linux glibc 2.28+ ARM64 | Details for kvbm-0.7.0-cp310-abi3-manylinux_2_28_aarch64.whl |
Total release size: 16.2 MB
Release files / kvbm-0.7.0-cp310-abi3-manylinux_2_28_x86_64.whl
| Download URL | kvbm-0.7.0-cp310-abi3-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 8.2 MB |
| Tags | CPython 3.10 Linux glibc 2.28+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
34af9f6a2008cbe5ae754273060d11bc10b5e8a44e8745310c45ccd4b4dc3bad
|
|
BLAKE2b-256 checksum How to use checksums |
e0dae0bbebe9c7dd085f7320b2de36d3e67df7ca442ac1c3b739a173ded3cef3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.0
|
Release files / kvbm-0.7.0-cp310-abi3-manylinux_2_28_aarch64.whl
| Download URL | kvbm-0.7.0-cp310-abi3-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 8.0 MB |
| Tags | CPython 3.10 Linux glibc 2.28+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
51f8e16b48133d0f600740b9c1e9ec50197e72417783e2366cb5aed175d01ad8
|
|
BLAKE2b-256 checksum How to use checksums |
8740fbc44c59718b20c523c2b5449fd310bc1b40a9b8fdbb55b865e1f0e926c4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.0
|
Security for kvbm 0.7.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.