Skip to main content

memkv-sglang

sglang HiCacheStorage backend that persists prefix KV pages in a remote MemKV cluster. Loaded as a vendor plugin via sglang's built-in dynamic storage backend dispatch — no patches to sglang's tree.

Build

cd sglang-plugin
pip install maturin
maturin develop --release      # local dev install
# or
maturin build --release        # wheel under target/wheels/
pip install target/wheels/memkv_sglang-*.whl

The wheel bundles a native PyO3 extension built from the same memkv-client crate the NIXL plugin uses, so RDMA/TCP transport selection works the same way.

Configure the MemKV connection

The plugin reads the standard MemKV config chain — MEMKV_CONFIG yaml first, then MEMKV_* env vars:

export MEMKV_SERVERS="10.0.0.10:9900,10.0.0.11:9900"
export MEMKV_RDMA_DEVICES="mlx5_0,mlx5_1"
export MEMKV_AUTH_KEY="<64-hex>"
export MEMKV_LICENSE="/etc/memkv/minio.license"   # one way to supply the license
# optional:
# export MEMKV_TRANSPORT=auto       # rdma | tcp | auto (default)
# export MEMKV_CONFIG=/etc/memkv.yaml

A license is required: the client verifies one when it builds its engine, and a plugin with no license available fails at startup with no license found rather than degrading. It is looked up in this order: the license: field of MEMKV_CONFIG, then MEMKV_LICENSE (an inline JWT or a path to a file holding one), then MINIO_LICENSE / AISTOR_LICENSE and the standard minio.license file locations. Contact MinIO to obtain one.

Launch sglang against MemKV

Use sglang's dynamic storage backend, pointing it at the class in this package:

python -m sglang.launch_server \
    --model-path meta-llama/Llama-3-8B \
    --enable-hierarchical-cache \
    --hicache-storage-backend dynamic \
    --hicache-storage-backend-extra-config '{
      "backend_name": "memkv",
      "module_path": "memkv_sglang.backend",
      "class_name": "MemKVHiCacheStorage"
    }'

sglang's StorageBackendFactory._create_dynamic_backend imports the class and constructs it as MemKVHiCacheStorage(storage_config, kwargs).

What's implemented

Method Status
get / batch_get yes; RDMA zero-copy direct into the target tensor when eligible (Linux + CPU + contiguous), bytes path otherwise
set / batch_set yes
exists / batch_exists yes (router-aware, one batched RPC per server)
batch_exists_v2 yes (per-pool hit policies)
batch_get_v2 yes; RDMA zero-copy into the dummy flat page when eligible
batch_set_v2 yes
clear no-op (server manages retention)

Layout

sglang-plugin/
├── Cargo.toml                   # cdylib + pyo3 + memkv-client
├── pyproject.toml               # maturin
├── src/lib.rs                   # PyO3 wrapper around memkv-client::Engine
└── python/memkv_sglang/
    ├── __init__.py              # re-exports Client
    └── backend.py               # MemKVHiCacheStorage(HiCacheStorage)

Metadata

Release files for memkv-sglang 1.0.16

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for memkv-sglang 1.0.16
File Interpreter ABI Platform Actions
memkv_sglang-1.0.16-cp38-abi3-manylinux_2_28_x86_64.whl CPython 3.8 abi3 Linux glibc 2.28+ x86-64 Details for memkv_sglang-1.0.16-cp38-abi3-manylinux_2_28_x86_64.whl
memkv_sglang-1.0.16-cp38-abi3-manylinux_2_28_aarch64.whl CPython 3.8 abi3 Linux glibc 2.28+ ARM64 Details for memkv_sglang-1.0.16-cp38-abi3-manylinux_2_28_aarch64.whl

Total release size: 3.0 MB

Release files / memkv_sglang-1.0.16-cp38-abi3-manylinux_2_28_x86_64.whl

Download URL memkv_sglang-1.0.16-cp38-abi3-manylinux_2_28_x86_64.whl
Size 1.5 MB
Tags CPython 3.8 Linux glibc 2.28+ x86-64 abi3
SHA-256 checksum
How to use checksums
ed541c3adb7f24c2880aa4ccb89522a387e9ef6bb56586fc929c442a63efb245
BLAKE2b-256 checksum
How to use checksums
d4686e3715d67581c0a43f5855fa5268af9c30da9cd466805005ed8aac64d838
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.7

Release files / memkv_sglang-1.0.16-cp38-abi3-manylinux_2_28_aarch64.whl

Download URL memkv_sglang-1.0.16-cp38-abi3-manylinux_2_28_aarch64.whl
Size 1.4 MB
Tags CPython 3.8 Linux glibc 2.28+ ARM64 abi3
SHA-256 checksum
How to use checksums
37357c1f335decae9488667a5367a28a4f0a79ffd52435a9aa54bf9db57bc62c
BLAKE2b-256 checksum
How to use checksums
b0ea14f79b8c7cdc96a24f598f9cbe01143a55544cb98896ba74a0bada0817ec
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.7

Security for memkv-sglang 1.0.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page