Skip to main content

memkv-sglang

sglang HiCacheStorage backend that persists prefix KV pages in a remote MemKV cluster. Loaded as a vendor plugin via sglang's built-in dynamic storage backend dispatch — no patches to sglang's tree.

Build

cd sglang-plugin
pip install maturin
maturin develop --release      # local dev install
# or
maturin build --release        # wheel under target/wheels/
pip install target/wheels/memkv_sglang-*.whl

The wheel bundles a native PyO3 extension built from the same memkv-client crate the NIXL plugin uses, so RDMA/TCP transport selection works the same way.

Configure the MemKV connection

The plugin reads the standard MemKV config chain — MEMKV_CONFIG yaml first, then MEMKV_* env vars:

export MEMKV_SERVERS="10.0.0.10:9900,10.0.0.11:9900"
export MEMKV_RDMA_DEVICES="mlx5_0,mlx5_1"
export MEMKV_AUTH_KEY="<64-hex>"
export MEMKV_LICENSE="/etc/memkv/minio.license"   # one way to supply the license
# optional:
# export MEMKV_TRANSPORT=auto       # rdma | tcp | auto (default)
# export MEMKV_CONFIG=/etc/memkv.yaml

A license is required: the client verifies one when it builds its engine, and a plugin with no license available fails at startup with no license found rather than degrading. It is looked up in this order: the license: field of MEMKV_CONFIG, then MEMKV_LICENSE (an inline JWT or a path to a file holding one), then MINIO_LICENSE / AISTOR_LICENSE and the standard minio.license file locations. Contact MinIO to obtain one.

Launch sglang against MemKV

Use sglang's dynamic storage backend, pointing it at the class in this package:

python -m sglang.launch_server \
    --model-path meta-llama/Llama-3-8B \
    --enable-hierarchical-cache \
    --hicache-storage-backend dynamic \
    --hicache-storage-backend-extra-config '{
      "backend_name": "memkv",
      "module_path": "memkv_sglang.backend",
      "class_name": "MemKVHiCacheStorage"
    }'

sglang's StorageBackendFactory._create_dynamic_backend imports the class and constructs it as MemKVHiCacheStorage(storage_config, kwargs).

What's implemented

Method Status
get / batch_get yes; RDMA zero-copy direct into the target tensor when eligible (Linux + CPU + contiguous), bytes path otherwise
set / batch_set yes
exists / batch_exists yes (router-aware, one batched RPC per server)
batch_exists_v2 yes (per-pool hit policies)
batch_get_v2 yes; RDMA zero-copy into the dummy flat page when eligible
batch_set_v2 yes
clear no-op (server manages retention)

Layout

sglang-plugin/
├── Cargo.toml                   # cdylib + pyo3 + memkv-client
├── pyproject.toml               # maturin
├── src/lib.rs                   # PyO3 wrapper around memkv-client::Engine
└── python/memkv_sglang/
    ├── __init__.py              # re-exports Client
    └── backend.py               # MemKVHiCacheStorage(HiCacheStorage)

Metadata

Release files for memkv-sglang 1.0.12

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for memkv-sglang 1.0.12
File Interpreter ABI Platform Actions
memkv_sglang-1.0.12-cp38-abi3-manylinux_2_28_x86_64.whl CPython 3.8 abi3 Linux glibc 2.28+ x86-64 Details for memkv_sglang-1.0.12-cp38-abi3-manylinux_2_28_x86_64.whl
memkv_sglang-1.0.12-cp38-abi3-manylinux_2_28_aarch64.whl CPython 3.8 abi3 Linux glibc 2.28+ ARM64 Details for memkv_sglang-1.0.12-cp38-abi3-manylinux_2_28_aarch64.whl

Total release size: 3.0 MB

Release files / memkv_sglang-1.0.12-cp38-abi3-manylinux_2_28_x86_64.whl

Download URL memkv_sglang-1.0.12-cp38-abi3-manylinux_2_28_x86_64.whl
Size 1.5 MB
Tags CPython 3.8 Linux glibc 2.28+ x86-64 abi3
SHA-256 checksum
How to use checksums
b110362983e7e7fd80e201aea9ec31db6268f457856de79d3c9bb5518da3e0df
BLAKE2b-256 checksum
How to use checksums
cdefc96131a5b83f31590bf002a28d05a8029e468f5e58e0c2d72de278f4d630
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.7

Release files / memkv_sglang-1.0.12-cp38-abi3-manylinux_2_28_aarch64.whl

Download URL memkv_sglang-1.0.12-cp38-abi3-manylinux_2_28_aarch64.whl
Size 1.4 MB
Tags CPython 3.8 Linux glibc 2.28+ ARM64 abi3
SHA-256 checksum
How to use checksums
0e1bb3bfd307dc849b55958562e4262452c6fe3b81a704a27c284f591db42ee2
BLAKE2b-256 checksum
How to use checksums
eb92d85c6c1340dcf65542bf08d906875ab172b6b756ea794c8ba4e6a7ddb97d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.7

Security for memkv-sglang 1.0.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page