SGLang filesystem linker

Implement UnifiedCacheLinker in a separately installed Python package and select its class and module at launch. You do not need to register the class inside SGLang or modify its backend selection code. This follows the explicit class/module/extra-options pattern of vLLM's KV connector configuration.

Install and launch

Use Linux with an NVIDIA CUDA GPU and a SGLang environment with the external linker loader from SGLang PR #40595. This package does not install or replace SGLang or its CUDA-specific PyTorch runtime. No tagged SGLang release is claimed to support the loader yet. The tested SGLang revision is 9f417bd5054f3ed2db54eeb07c741df6ff7c255c. The example uses one GPU (TP=DP=PP=CP=1), ordinary MHA/GQA in NHD layout, and float16, bfloat16, or float32 KV. SWA, MLA, Mamba, quantized KV, unified memory, and speculative decoding are intentionally unsupported by this example.

Install the independently maintained package into that environment:

python -m pip install sglang-fs-linker==0.1.0
python -c 'from sglang_fs_linker.linker import FileSystemLinker; print(FileSystemLinker.__module__)'

To install from this repository instead, run python -m pip install . from its root. SGLang itself does not contain or register this package.

Start SGLang with a public model (no Hugging Face token required):

python -m sglang.launch_server \
  --model-path Qwen/Qwen3-0.6B \
  --revision c1899de289a04d12100db370d81485cdf75e47ca \
  --host 127.0.0.1 --port 30000 \
  --tp 1 --page-size 16 --attention-backend triton \
  --disable-cuda-graph --mem-fraction-static 0.3 \
  --max-total-tokens 4096 --enable-cache-report \
  --enable-unified-cache-external-linker \
  --unified-cache-external-linker-config '{
    "linker": "FileSystemLinker",
    "linker_module_path": "sglang_fs_linker.linker",
    "linker_extra_config": {
      "directory": "/tmp/sglang-fs-kv",
      "namespace": "qwen3-0.6b-example-v1"
    }
  }'

The JSON has two required strings, linker and linker_module_path, and an optional object, linker_extra_config. Install the plugin and its dependencies in every worker environment. Only an explicitly selected module is imported, inside the cache-owning worker; frontend configuration validation does not import it. Use trusted plugin packages, since importing them executes Python.

This config takes precedence over --unified-cache-external-linker-backend. Without the JSON config, existing mooncake (default) and mori launches keep their constructor and behavior. The enable flag is still required. The Mooncake-specific --enable-linker-mla-dedup cannot be used with this config. There is no kv_role: this interface provides both lookup/load and offload.

Implement a linker

Subclass sglang.srt.mem_cache.unified_cache.unified_cache_linker.UnifiedCacheLinker and provide this constructor:

def __init__(self, server_args, params, *, components, extra_config):
    ...

params is CacheInitParams, including the worker's device allocator and rank groups. components is a set of active ComponentType values. extra_config is a private copy of the JSON options. Validate your supported model layouts and topology before allocating resources. The abstract interface documents all required methods, including reset and close.

PoolTransfer.device_indices identifies the GPU slots; keys identifies the corresponding prefix-hashed pages. A load must wait for start_layer_wise_loading() before copying into those slots. Publish completion only after device writes are safe, because the tree releases locks on completion. The layer_done_counter provides set_consumer(index) and wait_until(layer) to the worker and KV pool. Asynchronous implementations must synchronize each layer correctly; this example finishes all copies synchronously before returning.

SGLang continues to own prefix matching, GPU allocation, admission and tree locks. The plugin owns storage and transfer completion. Existing Full/SWA component restrictions in the tree wrapper still apply; the plugin mechanism does not extend model support by itself.

Verify disk reuse

With the server above running, use another shell in the same environment:

sglang-fs-linker-verify --base-url http://127.0.0.1:30000

The verifier sends a cold request with a unique cache salt, flushes the GPU radix cache, then repeats the request. It requires a positive external hit in meta_info.cached_tokens_details.host, zero device-cache hits, identical output tokens, and output log probabilities within 0.05. The server also reports FileSystemLinker loaded N pages from disk. A warm request without a radix flush would only demonstrate local prefix caching.

Development and verification

Clone both repositories into a parent directory. Check out the SGLang revision with the loader; the FS package stays outside that checkout:

git clone --branch feat/out-of-tree-unified-cache-linker https://github.com/maobaolong/sglang.git
git -C sglang checkout 9f417bd5054f3ed2db54eeb07c741df6ff7c255c
git clone https://github.com/maobaolong/sglang-fs-linker.git

For the NVIDIA H200 validation environment, use Docker with NVIDIA Container Toolkit. Run this from the parent directory containing both repositories:

docker run --rm -it --gpus all --ipc=host \
  -v "$PWD:/work" -w /work \
  --entrypoint bash \
  lmsysorg/sglang@sha256:9e148f5ac788e856a06166bd6347a831831eb9fcfab4d1770874823a7c29a1a1

Inside the container, select one available GPU and prepare the runtime:

export CUDA_VISIBLE_DEVICES=0
export PYTHONPATH=/work/sglang/python
python -m pip install --no-deps sglang-kernel==0.4.7
python -m pip install /work/sglang-fs-linker
python -c 'import sglang; print(sglang.__file__)'
python -m pytest -q /work/sglang-fs-linker/tests

The printed SGLang path must belong to the mounted checkout. The image provides Python 3.12 and PyTorch 2.13.0+cu130, but its preinstalled kernel package is older than the tested checkout requires. No native rebuild is needed. Run the launch command above; run the installed verifier in a second shell in that container.

The filesystem tests use real CPU tensors, temporary files, and separately constructed producer/consumer instances. They check exact KV restoration into different slots, persistence across reset/close, failed writes, missing prefixes, cancellation, and corrupt pages. No Mooncake or MORI service is needed.

The SGLang loader and existing linker regressions remain in SGLang. In the same container, run them separately:

cd /work/sglang
python -m pytest -q \
  test/registered/unit/mem_cache/test_external_linker_factory.py \
  test/registered/unit/mem_cache/test_registry.py \
  test/registered/unit/server_args/test_server_args.py \
  test/registered/unit/mem_cache/test_unified_cache_linker.py \
  test/registered/unit/mem_cache/test_linker_pool_assembler.py

For packaging-only checks (no GPU or SGLang required), run these commands from the sglang-fs-linker repository root:

uv build
uvx twine check --strict dist/*
uvx --from ruff==0.15.1 ruff check src tests
uvx --from ruff==0.15.1 ruff format --check src tests

Filesystem example limits

Each file contains all K/V layers of one page. A temporary file followed by atomic replacement keeps incomplete pages invisible. Reset and close preserve files, so the cache can survive worker restarts. The namespace, model path, page size, dtype, and tensor geometry isolate files; change namespace whenever model weights, revision, adapters, or RoPE settings change. Do not reuse a namespace across incompatible model configurations.

This is a validation example, not a performance backend: transfers synchronize the device, stage through CPU tensors, and perform blocking filesystem I/O. It has no disk eviction, capacity limit, or crash-durability guarantee. Remove its directory only when its workers are stopped. Missing files are misses before a load is queued; deletion or corruption after admission fails the load instead of allowing inference on incomplete KV. Only use a directory managed by trusted processes.