Skip to main content

memkv-sglang

sglang HiCacheStorage backend that persists prefix KV pages in a remote MemKV cluster. Loaded as a vendor plugin via sglang's built-in dynamic storage backend dispatch — no patches to sglang's tree.

Build

cd sglang-plugin
pip install maturin
maturin develop --release      # local dev install
# or
maturin build --release        # wheel under target/wheels/
pip install target/wheels/memkv_sglang-*.whl

The wheel bundles a native PyO3 extension built from the same memkv-client crate the NIXL plugin uses, so RDMA/TCP transport selection works the same way.

Configure the MemKV connection

The plugin reads the standard MemKV config chain — MEMKV_CONFIG yaml first, then MEMKV_* env vars:

export MEMKV_SERVERS="10.0.0.10:9900,10.0.0.11:9900"
export MEMKV_RDMA_DEVICES="mlx5_0,mlx5_1"
export MEMKV_AUTH_KEY="<64-hex>"
export MEMKV_LICENSE="/etc/memkv/minio.license"   # one way to supply the license
# optional:
# export MEMKV_TRANSPORT=auto       # rdma | tcp | auto (default)
# export MEMKV_CONFIG=/etc/memkv.yaml

A license is required: the client verifies one when it builds its engine, and a plugin with no license available fails at startup with no license found rather than degrading. It is looked up in this order: the license: field of MEMKV_CONFIG, then MEMKV_LICENSE (an inline JWT or a path to a file holding one), then MINIO_LICENSE / AISTOR_LICENSE and the standard minio.license file locations. Contact MinIO to obtain one.

Launch sglang against MemKV

Use sglang's dynamic storage backend, pointing it at the class in this package:

python -m sglang.launch_server \
    --model-path meta-llama/Llama-3-8B \
    --enable-hierarchical-cache \
    --hicache-storage-backend dynamic \
    --hicache-storage-backend-extra-config '{
      "backend_name": "memkv",
      "module_path": "memkv_sglang.backend",
      "class_name": "MemKVHiCacheStorage"
    }'

sglang's StorageBackendFactory._create_dynamic_backend imports the class and constructs it as MemKVHiCacheStorage(storage_config, kwargs).

What's implemented

Method Status
get / batch_get yes; RDMA zero-copy direct into the target tensor when eligible (Linux + CPU + contiguous), bytes path otherwise
set / batch_set yes
exists / batch_exists yes (router-aware, one batched RPC per server)
batch_exists_v2 yes (per-pool hit policies)
batch_get_v2 yes; RDMA zero-copy into the dummy flat page when eligible
batch_set_v2 yes
clear no-op (server manages retention)

Layout

sglang-plugin/
├── Cargo.toml                   # cdylib + pyo3 + memkv-client
├── pyproject.toml               # maturin
├── src/lib.rs                   # PyO3 wrapper around memkv-client::Engine
└── python/memkv_sglang/
    ├── __init__.py              # re-exports Client
    └── backend.py               # MemKVHiCacheStorage(HiCacheStorage)

Metadata

Release files for memkv-sglang 1.0.14

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for memkv-sglang 1.0.14
File Interpreter ABI Platform Actions
memkv_sglang-1.0.14-cp38-abi3-manylinux_2_28_x86_64.whl CPython 3.8 abi3 Linux glibc 2.28+ x86-64 Details for memkv_sglang-1.0.14-cp38-abi3-manylinux_2_28_x86_64.whl
memkv_sglang-1.0.14-cp38-abi3-manylinux_2_28_aarch64.whl CPython 3.8 abi3 Linux glibc 2.28+ ARM64 Details for memkv_sglang-1.0.14-cp38-abi3-manylinux_2_28_aarch64.whl

Total release size: 3.0 MB

Release files / memkv_sglang-1.0.14-cp38-abi3-manylinux_2_28_x86_64.whl

Download URL memkv_sglang-1.0.14-cp38-abi3-manylinux_2_28_x86_64.whl
Size 1.5 MB
Tags CPython 3.8 Linux glibc 2.28+ x86-64 abi3
SHA-256 checksum
How to use checksums
0f4a2cf24aa71362103a1653c1bad05b00a041ff65139885fafa28609e6e8fe8
BLAKE2b-256 checksum
How to use checksums
af2c7d1ca27d1532ab50035ea11db2e8e747a3e4462fd130b0b9501e35c68b0d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.7

Release files / memkv_sglang-1.0.14-cp38-abi3-manylinux_2_28_aarch64.whl

Download URL memkv_sglang-1.0.14-cp38-abi3-manylinux_2_28_aarch64.whl
Size 1.4 MB
Tags CPython 3.8 Linux glibc 2.28+ ARM64 abi3
SHA-256 checksum
How to use checksums
fcc1a62b70acf000f23f202fcf9132a02b1f2b813b56e24515f7619af3f531f3
BLAKE2b-256 checksum
How to use checksums
f4d4c5e512b570823b9cb8ce4b99effbaeeb1a3b0c7fa64aaf676f5aedb02574
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.7

Security for memkv-sglang 1.0.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page