autumn-kvcache
Daemon-less KV-cache L3 storage backends built on the autumn-rs partition
layer. Two adapters, one data plane (autumn.BatchClient + the sync/async
_bridge):
| Adapter | Engine | Contract | Module |
|---|---|---|---|
AutumnKVCacheStorage |
sglang | HiCache HiCacheStorage (L3) |
autumn_kvcache.sglang_backend |
AutumnKVConnector |
vLLM | KVConnectorBase_V1 (external KV transfer) |
autumn_kvcache.vllm_connector |
Which autumn-rs this tracks
AUTUMN_REV names the autumn-rs commit this tree was last verified against.
It matters more than a normal version pin: the adapters ride autumn's rkyv
client, and a client/cluster mismatch does not fail the handshake — it can
decode garbage. FreeToken's docker/Dockerfile.freetoken-l3 carries the same
pin as AUTUMN_IMAGE_PIN for exactly this reason.
The PyO3 extension (import autumn) is the autumn-rs distribution on PyPI:
one manylinux abi3 wheel per architecture for every CPython >= 3.11. Its
version must match the cluster's autumn-rs release; for a cluster built from
another commit, build the extension from that commit (below).
There is no daemon and no local DRAM cache — the autumn partition layer is
the only persistence path. KV pages are content-addressed
(kvc/{h}/{model_scope}/{pool}/{hash}/{layer}, h = the hash's first hex
char), so there is no invalidation. The cluster serves them once the kvc
namespace exists: autumn-op namespace-create --name kvc --parts 16 registers
it and cuts it at kvc/<hex> (autumn-rs cluster.sh does this on its first
start). h comes first so those 16 partitions take every model's pages with
no per-model step.
Install
pip install autumn-kvcache # pulls autumn-rs, the native `autumn` client
pip install "git+https://github.com/journeymidnight/autumn_kvcache@<rev>" # or from git
The native client from source, for a cluster at another commit or for RDMA (an autumn-rs checkout at that commit):
cd autumn-rs/python
maturin build --release --interpreter /path/to/venv/bin/python
# for UCX: add `--features ucx --auditwheel skip`; the wheel must link the
# system UCX, since auditwheel's bundled copy has no mlx5 transport modules
# and the client then cannot connect over RDMA
/path/to/venv/bin/pip install --force-reinstall target/wheels/autumn_rs-*.whl
Both adapter modules import without the engine (sglang / vLLM) installed —
the engine import is defensive — so python -c "from autumn_kvcache.vllm_connector import AutumnKVConnector" works for tooling and
the data-plane smoke test regardless.
Faster reads: the partition server's memory tier
Start the partition servers with --mem-tier-bytes <budget> --mem-tier-prefix kvc/
and KV pages are held in their DRAM and acknowledged without a log write; reads
come from memory. Pages held there are lost on a PS restart (a cache miss, then
recompute). Measured with SGLang + Qwen3-VL-32B: per-request p50 1344 → 980 ms,
2 GiB fetch 999 → 699 ms. See autumn-rs docs/ops.md ("Memory tier for KV cache").
The tier evicts least recently used first, and among pages used at about the
same time the deeper ones first, so a prefix loses its tail before its head
(a hit is a run of pages from the start). The adapters pass each page's
position in its prefix with the write (evict_rank, autumn wire 87): vLLM the
prefix's length; sglang (pages before the batch + page index) × page_size.
sglang tells the backend the pages before a batch only with
"hicache_storage_pass_prefix_keys": true in
--hicache-storage-backend-extra-config; set it. Without it every batch ranks
from 0, and even with it a node written from the root over several 128-page
batches gets an empty list after the first (sglang only extends a non-empty
one): those batches rank from 0 too, which orders them by write time alone.
The tier can evict a prefix's layers while its vLLM __present__ marker stays.
A load that then finds a layer missing injects nothing, and the connector
deletes that marker (the worker its own rank's, the scheduler the one it
probes), so the next request for the prefix saves it again. Run vLLM with
kv_load_failure_policy="recompute": under the default "fail" the request
whose load failed ends with an error instead of being recomputed.
Using autumn-kvcache as a vLLM L3 (cross-instance prefix cache)
AutumnKVConnector is a native vLLM V1 KV connector (Phase 3a — CPU-offload
path). It offloads each request's block-aligned prompt-prefix KV into autumn on
the producing instance, and loads it back on any instance that later sees
the same prefix — a cross-instance / cross-restart prefix cache backed by the
partition layer, with no extra service to run.
Pinned vLLM: verified against vLLM 0.23.0 (torch 2.11). The
KVConnectorBase_V1signatures drift across vLLM releases (e.g. the external connector constructor gained a 3rdkv_cache_configargument); the adapter forwards it when present and falls back to the 2-arg form, but pin a known version in production.
Point both engines at a running autumn cluster's manager endpoint and give
each vllm serve the connector config:
KVCFG='{
"kv_connector":"AutumnKVConnector",
"kv_connector_module_path":"autumn_kvcache.vllm_connector",
"kv_role":"kv_both",
"kv_connector_extra_config":{"endpoint":"127.0.0.1:9001","transport":"tcp"}
}'
# instance 1 (e.g. GPU 0)
CUDA_VISIBLE_DEVICES=0 vllm serve /models/Qwen3-8B --served-model-name qwen3 \
--port 8101 --enforce-eager --kv-transfer-config "$KVCFG"
# instance 2 (e.g. GPU 1) — a SEPARATE engine, empty local prefix cache
CUDA_VISIBLE_DEVICES=1 vllm serve /models/Qwen3-8B --served-model-name qwen3 \
--port 8102 --enforce-eager --kv-transfer-config "$KVCFG"
kv_connector_extra_config keys: endpoint (autumn manager host:port,
required), transport (tcp default, or ucx), max_inflight, ttl_secs,
model_id (optional explicit model identity — see below). The client reads
with one Rust worker: ~5 GB/s per rank on TCP, ~42 GB/s on UCX (rendezvous
into the page); client_workers is no longer read.
What the external cache does (and does not) speed up
The connector is an L3 behind vLLM's own local prefix cache (GPU + host RAM). vLLM matches the local cache first and asks the connector only for the tokens beyond the local match, because remote storage is slower than a local hit — you would never want to prefer autumn over GPU cache. Concretely:
- Same engine, repeated prompt → the local prefix cache serves it, so the external cache is (correctly) never loaded and its hit rate is ~0%. This is expected, not a failure.
- A different engine / a restarted engine / after local eviction → the local cache is cold, so the connector loads the prefix from autumn and skips prefill. This is the whole point (measured: a 1.3 k-token prefix loads in ~0.3 s vs. ~2 s of recompute — a ~3–4× TTFT win on the cold-local path).
So judge the connector by cross-instance / post-restart hit rate, not by same-instance repeats. It also never re-saves a prefix that is already durable in autumn (BUG-KVC-NO-HIT): a repeat whose KV the local cache already served writes nothing, so it costs neither storage nor prefill time. (It issues one lightweight presence probe per new request to make that de-dup decision.)
Saves are asynchronous (almost nothing on the prefill critical path)
On the forward pass, save_kv_layer does only the cheap GPU-side gather
(_extract_layer → a standalone tensor, decoupled from the paged blocks — no
CPU sync). Everything expensive runs on a background thread: the D2H .cpu()
copy, the store-dedup probe, the durable batched put_from, and the marker. A
CUDA event lets the background thread wait for the gathers before the D2H, and
the __present__ marker is published only after every layer ACKs (so a reader
that sees the marker always finds a complete, correct prefix — verified: an
external-load reproduces vLLM's local-cache output token-for-token).
Measured no-hit overhead (Qwen3-VL-32B, 1 k-token prompt, H200): TTFT +≈6 ms, TPOT ≈0 vs. no connector — i.e. a prefill that never gets reused costs almost nothing. (Before moving the D2H off the path it was +≈148 ms.) When the prefix is reused cross-instance, the connector skips the whole prefill instead.
A prompt longer than max_num_batched_tokens is prefilled in chunks over
several steps. Each step gathers only the tokens it computed (so the GPU staging
stays within vLLM's token budget), the background thread copies them into the
prefix's host buffer, and the prefix is written once its last chunk is in. A
layer's value larger than the partition server's 64 MiB cap (Qwen3-VL-4B: a
prefix over 16K tokens) is stored as equal pieces {layer}/{i}.
At most _MAX_INFLIGHT_SAVES saves are in flight, from a prefix's first chunk to
its write; each holds a registered host buffer the size of its prefix's KV, and
the buffers grow to the longest prefix seen (at most 2 × max_model_len ×
per-token KV). Over that a save is dropped rather than blocking prefill — it is
a pure cache, so a later request re-saves. The store-dedup also lives in the
background: a prefix already durable in autumn is neither re-copied nor
re-written, so a repeated prompt (served by the local GPU cache) costs no storage
and no extra latency.
Model-scope isolation: the model's real identity is part of the key
The model-scope segment of every key is
{model}_{fingerprint}_{tp_rank}_{tp_size}[...], where fingerprint is a
short hash of the model's identity: architecture shape
(layers / hidden / kv-heads / head-size / vocab / dtype / quantization / MLA)
plus the weights source (load_format, and for --load-format autumn the
autumn weights path from model_loader_extra_config), plus model_id if
you set one, plus the layout versions — the running vLLM version
(full x.y.z: the KV page layout is a vLLM-internal detail with no stability
contract, patch releases included) and the connector's own storage-format
version (VLLM_KV_STORAGE_FORMAT, bumped whenever the extract/inject byte
layout changes).
This exists because the model path is not an identity: with
a shared config dir, every model is served from the same fixed local config
dir, so two different models used to share one model scope and cross-read each
other's KV (observed live: Qwen2.5-7B and 32B under one model scope; a same-shape
pair would have been silently wrong). Set
kv_connector_extra_config["model_id"] when even the fingerprint can't
distinguish your deployments — e.g. two finetunes with identical architecture
loaded from the same path, or weights overwritten in place at one autumn
path (don't do that — store new weights at a new path).
Upgrade note: any change to the fingerprint inputs — including every vLLM version change, patch releases too — moves the model scope, so the whole vLLM pool goes cold and rebuilds on first use (pure cache, content-addressed — no migration needed). This is deliberate: an upgrade already restarts the pods (GPU cache is lost anyway), so the re-warm is a one-time, predictable cost, whereas carrying KV across a layout-incompatible vLLM would be silent garbage. Keys written by older connectors stay behind under the old model scope; with
ttl_secs=0they never expire, so reclaim them manually if you care about the space:client.batch_delete(f"kvc/{h}/<old-model-scope>/vllm/".encode())for eachhin0123456789abcdef(keys from before the hash-bucket layout: the single prefixkvc/<old-model-scope>/; the old model scope is in the previous deployment's startup log).
ttl_secs (default 0 = no expiry) is the relative TTL after which an
offloaded prefix stops being served. Content-addressed keys never invalidate,
so a TTL is the only reclamation knob for a long-running cluster. Two caveats on
what exactly it bounds:
- The connector writes the per-prefix completion marker with
ttl_secsand the layer pages withttl_secs + grace(grace = 300 s), so the marker always expires before its layers. The scheduler admits a load on the marker; a marker that outlived its layers would let the worker silently load uninitialised KV (a correctness bug). Sottl_secsbounds admissibility (when the prefix can still be hit); the layer data lingers ~gracelonger before its own lazy expiry. - Reclamation is lazy (keys are dropped on read after they expire, plus background compaction), not a hard wall-clock free — size capacity planning accordingly.
Verifying the cross-instance hit
Send a long prompt (longer than the cache block size) to instance 1, then the same prompt to instance 2. Instance 2 has never seen it locally, so any external-cache hit is served from autumn:
curl -s localhost:8101/v1/completions -H 'Content-Type: application/json' \
-d '{"model":"qwen3","prompt":"<long prompt>","max_tokens":1}' >/dev/null
curl -s localhost:8102/v1/completions -H 'Content-Type: application/json' \
-d '{"model":"qwen3","prompt":"<same long prompt>","max_tokens":1}' >/dev/null
# instance 2's connector metrics now show the cross-instance hit:
curl -s localhost:8102/metrics | grep -E \
'vllm:external_prefix_cache_hits_total|external_kv_transfer'
# vllm:external_prefix_cache_hits_total{...} 272.0
# vllm:prompt_tokens_by_source_total{source="external_kv_transfer"} 272.0
(vLLM 0.23's OpenAI usage does not surface cached_tokens for the
connector path — use the vllm:external_prefix_cache_* /
prompt_tokens_by_source{source="external_kv_transfer"} metrics, which are the
authoritative external-hit signal.)
A scripted version of this check is tests/ + the e2e in the repo.
Data-plane smoke test (no engine required)
Exercises the autumn-facing core (_AutumnKVStore) against a real 1-node
cluster — key format, byte-exact page store/load round-trip, existence probe:
AUTUMN_KVCACHE_ENDPOINT=127.0.0.1:9001 AUTUMN_KVCACHE_TRANSPORT=tcp \
python -m tests.test_vllm_dataplane # run from this repo root
Architecture notes
- Stateless adapter / content-addressed keys — no invalidation, partition is the only persistence path.
- TTL reclamation (
ttl_secs) — both adapters acceptttl_secsin their extra-config (vLLMkv_connector_extra_config, sglanghicache-storage-backend-extra-config); default0= no expiry. The partition layer expires keys lazily on read. The sglang backend just stamps every page (its own L1/L2 + hash manage existence, so an expired L3 key is a clean miss); the vLLM connector additionally keeps the completion marker's TTL shorter than its layers' (see above) to preserve load correctness. - sglang host-pool layouts — a page is read into, and written from, the
host pool itself. Where the page (MLA
page_first*) or each of its K/V halves (MHApage_first*) is a window into the pool, it moves with no copy; otherwise (layer_first, the default, and pools that assemble a page, such as Mamba) it is copied out to write, and a read lands in a scratch buffer thatset_from_flat_data_pageputs into the pool. Stored bytes follow the pool's own order, so every layout butlayer_firstcarries its name in the key: instances with different layouts never read each other's pages. For the copy-free path, run sglang with--hicache-mem-layout page_first_direct; the copy path keeps a scratch buffer the size of one batch (128 pages) on each of sglang's backup and prefetch threads. These layouts need the v1 interface ("interface_v1": 1); the v0 wrappers use the untagged key. - No duplicate writes — a page already stored (all of its values present)
is not written again; one existence probe per batch. Its
ttl_secstherefore counts from the first write: backing a hot page up again does not extend it. - Return fast — never block past the engine's step budget; the load path is synchronous in Phase 3a (per-layer overlap is Phase 3b).
- Model-scope isolation — keys carry a model scope derived from model +
model-identity fingerprint (
_identity.py) + TP/PP, so different models / parallel layouts never alias. The sglang model-scope format is unchanged (model_nameis a real identity there); its escape hatch isextra_config["model_id"].
Offline unit tests (no cluster, no engine, no native module)
uv run --with pytest python -m pytest tests/test_model_identity.py -q # from this repo root
python -m pytest tests/test_page_layouts.py -q # needs torch; every sglang host-pool layout
Metadata
Release files for autumn-kvcache 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| autumn_kvcache-0.1.0.tar.gz | 90.6 kB | Without provenance | Details for autumn_kvcache-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| autumn_kvcache-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for autumn_kvcache-0.1.0-py3-none-any.whl |
Total release size: 149.9 kB
Release files / autumn_kvcache-0.1.0.tar.gz
| Download URL | autumn_kvcache-0.1.0.tar.gz |
|---|---|
| Size | 90.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c296c0408d07af0ebe59103ce749880e69c95b6def3aa45f8536f468a3579ded
|
|
BLAKE2b-256 checksum How to use checksums |
54edb6bd1cedac68f26537f76b583865aee3fd0672295bdf4edb90710d60f74a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.26 {"installer":{"name":"uv","version":"0.11.26","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / autumn_kvcache-0.1.0-py3-none-any.whl
| Download URL | autumn_kvcache-0.1.0-py3-none-any.whl |
|---|---|
| Size | 59.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
73202ee84428d6dd378c1de987c6e358fd6dc49d3f45b38d2b10312634db0334
|
|
BLAKE2b-256 checksum How to use checksums |
cb20860d3885bc972a30eaf032029642cceb6c0b14c09af3c440a487fa3039df
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.26 {"installer":{"name":"uv","version":"0.11.26","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for autumn-kvcache 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.