telys-vllm
An open-source external KV / prefix-cache connector for vLLM
(KVConnectorBase_V1) backed by Telys: restart-persistent,
cross-engine KV cache that provably beats recomputation on long shared prefixes.
The closed Telys engine stays closed — it is fetched by telys login and runs
as a signed local runtime, like the telys SDK. This repository is the Apache-2.0
integration layer: the connector, tests, benchmarks, and telemetry that let vLLM
save and load KV blocks through Telys.
What it does
- Plugs into vLLM V1 as a custom
KVConnectorBase_V1. - Hashes per-block KV tokens and stores them in Telys so they survive process restarts and can be shared across replicas.
- On a cache hit, loads the stored blocks and injects them into vLLM's paged KV cache, skipping prefix recomputation.
- Ships with a fast path that stores stacked per-layer CPU tensors and uses batched, per-layer GPU injection.
Measured results
All numbers below come from bench/RESULTS.md. Workload: Qwen/Qwen2.5-7B-Instruct,
4096-token shared prefix, inputs 4096–8192 tokens, 64 output tokens, vLLM native
prefix caching OFF (--no-enable-prefix-caching), RunPod A40 (EU-SE-1),
telys==0.1.10, vllm==0.32.0.
| Version | Arm | TTFT p50 | Total time | Throughput | Speedup vs recompute |
|---|---|---|---|---|---|
| Autopsy v5 | Recompute (no cache) | 49.3 s | 106.8 s | 60.0 tok/s | 1.0× |
| Autopsy v5 | Telys warm hit | 27.8 s | 78.8 s | 81.2 tok/s | 1.8× TTFT, 1.35× total |
| Autopsy v6 | Recompute, 8K prefix | 89.3 s | 189.1 s | 33.9 tok/s | 1.0× |
| Autopsy v6 | Telys warm, 8K prefix | 25.1 s | 41.9 s | 152.7 tok/s | 3.6× TTFT, 4.5× total |
| Autopsy v6 | Recompute, 16K prefix | 186.6 s | 383.3 s | 16.7 tok/s | 1.0× |
| Autopsy v6 | Telys warm, 16K prefix | 48.6 s | 74.1 s | 86.4 tok/s | 3.8× TTFT, 5.2× total |
| Autopsy v7 | Recompute (fresh process) | 49.7 s | 103.9 s | 61.6 tok/s | 1.0× |
| Autopsy v7 | Telys warm, second replica | 16.8 s | 29.0 s | 220.9 tok/s | 2.96× TTFT, 3.58× total |
Key takeaways:
- The connector beats recomputation on total wall time at 4K and longer prefixes.
- The win grows with prefix length: 3.6× TTFT at 8K, 3.8× at 16K.
- The cross-replica / restart story works: a second, fresh vLLM process with an empty native cache warm-hits the Telys store populated by the first process.
When it helps / when it doesn't
| Scenario | Verdict | Why |
|---|---|---|
| Process restart or redeploy | ✅ Helps | KV blocks live in Telys, so a new process starts warm. |
| Multiple replicas / horizontal scale | ✅ Helps | Replicas share one Telys store; a cold replica warm-hits immediately. |
| Prefixes larger than native GPU KV budget | ✅ Helps | Telys persists blocks that vLLM's native cache would evict. |
| Single long-lived engine, prefix fits in GPU cache | ⚠️ Native caching wins | vLLM's in-memory prefix cache is faster when the engine never restarts and the KV budget is large enough. The connector adds serialization and persistence overhead that only pays off across restarts, replicas, or very long prefixes. |
| Short prefixes on small models | ❌ Not helpful | Recomputing a 512-token prefix on a 1.5B model is cheaper than loading it from an external store. |
Current status
Proof-level and working. The connector passes the local suite, correctly round-trips KV blocks, and the warm-hit path is measurably faster than recomputation on the workloads above. It is not yet a universal drop-in win: performance depends on prefix length, model size, GPU memory budget, and whether the deployment restarts or scales replicas.
- Local tests: 131 passing, 7 skipped (tests that require a live GPU or the full Telys runtime are skipped on headless runners).
- Supported vLLM:
>=0.11(tested most recently on0.32.0). - Supported Python:
>=3.10.
Install
pip install telys-vllm[vllm]
The [vllm] extra pulls in vllm>=0.11. The connector itself has no runtime
dependency on the closed Telys engine; the engine is installed separately by
pip install telys and telys login.
Usage
from vllm import LLM
llm = LLM(
model="Qwen/Qwen2.5-7B-Instruct",
kv_connector="telys_vllm.connector.TelysKVConnector",
kv_connector_extra_config={"telys_path": "./telys_kv"},
enable_prefix_caching=False, # let Telys be the cache of record
)
See bench/ for reproducible benchmark scripts and DESIGN.md for the
architecture and evolution of the connector.
Repository layout
telys_vllm/ # Connector source
connector.py # vLLM KVConnectorBase_V1 implementation
store.py # Telys-backed block store
codec.py # Raw-byte / stacked-tensor codecs
gpu_ops.py # GPU injection helpers
local_client.py # Slab / BlobStore client adapters
telemetry.py # JSONL instrumentation used by benches
tests/ # Unit and integration tests
bench/ # Benchmarks, run scripts, and RESULTS.md
Contributing
See CONTRIBUTING.md. The Telys engine runtime itself is closed and distributed separately; contributions here are limited to the open connector, tests, benchmarks, and docs.
License
Metadata
Release files for telys-vllm 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| telys_vllm-0.1.1-py3-none-any.whl | Python 3 | none | any | Details for telys_vllm-0.1.1-py3-none-any.whl |
Release files / telys_vllm-0.1.1-py3-none-any.whl
| Download URL | telys_vllm-0.1.1-py3-none-any.whl |
|---|---|
| Size | 47.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d8e4926cd89f37901ea0aeed73ebeb28010d42118c3dd6aeddb7e2cde87a6133
|
|
BLAKE2b-256 checksum How to use checksums |
9684831d44f6235e4a717785a29546169e9b794640c6ac0900a3dc7623c162bc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for telys-vllm 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.