Skip to main content

telys-vllm

An open-source external KV / prefix-cache connector for vLLM (KVConnectorBase_V1) backed by Telys: restart-persistent, cross-engine KV cache for shared-prefix workloads.

The closed Telys engine stays closed — it is fetched by telys login and runs as a signed local runtime, like the telys SDK. This repository is the Apache-2.0 integration layer: the connector, tests, benchmarks, and telemetry that let vLLM save and load KV blocks through Telys.

What it does

  • Plugs into vLLM V1 as a custom KVConnectorBase_V1.
  • Hashes per-block KV tokens and stores them in Telys so they survive process restarts and can be shared across replicas.
  • On a cache hit, loads the stored blocks and injects them into vLLM's paged KV cache, skipping prefix recomputation.
  • Ships with a fast path that stores stacked per-layer CPU tensors and uses batched, per-layer GPU injection.

Measured results

The current apples-to-apples concurrent matrix is run ca40-20261108T181727Z on one NVIDIA A40. It used Qwen/Qwen2.5-7B-Instruct, vLLM 0.32.0, BF16, the same direct token-ID prompts and greedy 64-token decoding in every arm, max_num_seqs=32, gpu_memory_utilization=0.95, and max_model_len=32768. Each cell has three repetitions. TTFT p50 pools all requests; wall and output throughput are means of the three batch-level values. Full settings, reproduction, and exact source provenance are in bench/CONCURRENT.md.

Prefix C Arm Pooled TTFT p50 (s) Mean wall (s) Mean output tok/s Wall speedup vs recompute
8K 1 Forced recompute 1.165793 2.985708 21.435457 1.0000x
8K 1 Fresh-process Telys external hit 1.104727 2.913860 21.974601 1.0247x
8K 1 Native-hot, same process 0.060614 1.878551 34.068840 1.5894x
8K 4 Forced recompute 3.969180 6.572369 38.950953 1.0000x
8K 4 Fresh-process Telys external hit 1.302441 3.343736 76.573539 1.9656x
8K 4 Native-hot, same process 0.117142 2.029236 126.156782 3.2388x
8K 8 Forced recompute 6.250244 11.383825 44.976216 1.0000x
8K 8 Fresh-process Telys external hit 1.548653 3.716415 138.050123 3.0631x
8K 8 Native-hot, same process 0.158475 2.146576 238.521740 5.3032x
16K 8 Forced recompute 12.908762 23.655112 21.644457 1.0000x
16K 8 Fresh-process Telys external hit 2.706828 5.396244 94.979480 4.3836x
16K 8 Native-hot, same process 0.252128 2.418234 211.732989 9.7820x

The 8K/c1 external result is approximately neutral (1.0247x mean-wall speedup; one repetition was 0.9992x). Three repetitions do not establish statistical significance.

Recompute and fresh-process external arms disable native prefix caching. External uses a separate drained writer and a fresh reader with a warm persistent store. Native-hot is a separate same-process arm, reusing GPU-resident native prefix KV after an excluded writer. Native-hot had lower mean wall and pooled TTFT p50 and higher output throughput than both other arms in every measured cell; its ratios are excluded from external persistent-cache speedups. Older sequential/no-cache v5-v7 results are retained in bench/RESULTS.md with their original baselines.

Validation

  • Independent A40 verification passed 25,414/25,414 16K blocks and 2,949/2,949 4K blocks. All A/B/C ordinary correctness and output-equality checks passed on both profiles. See the full profile measurements and limitations.
  • Current local tests and Ruff pass. Real Telys 0.1.14 / vLLM 0.32.0 A40 validation covered persistence, reopen, reset races, and two opt-in output-correctness passes. Bitwise restored-KV and 64-token output equality are validated with VLLM_BATCH_INVARIANT=1 and greedy decoding. See correctness scope and provenance.
  • Serverless lifecycle validation returned matching output from fresh workers; first lookups hit 538 blocks / 8,608 tokens, including after scale-to-zero. See lifecycle measurements and evidence.

Current status

Working connector with source-scoped validation. Performance is reported by workload, including batching, prefix length, model, GPU memory, and process or replica lifecycle.

  • Telys 0.1.15 adds durable BlobStore deletion tombstones. Use that SDK version or newer; the connector also rejects orphaned blob-only records on affected older stores instead of injecting deleted KV data.
  • Supported vLLM: >=0.11 (tested most recently on 0.32.0).
  • Supported Python: >=3.10.
  • Async loading is unsupported in this revision and fails fast; use synchronous loads (TELYS_VLLM_LOAD_ASYNC=0).

Install

pip install "telys>=0.1.15" "telys-vllm[vllm]"
telys login

The [vllm] extra pulls in vllm>=0.11. The connector itself has no runtime dependency on the closed Telys engine; telys login installs and verifies the signed on-device runtime.

Usage

from vllm import LLM

llm = LLM(
    model="Qwen/Qwen2.5-7B-Instruct",
    kv_connector="telys_vllm.connector.TelysKVConnector",
    kv_connector_extra_config={"telys_path": "./telys_kv"},
    enable_prefix_caching=False,  # let Telys be the cache of record
)

See bench/ for reproducible benchmark scripts and DESIGN.md for the architecture and evolution of the connector.

Repository layout

telys_vllm/      # Connector source
  connector.py   # vLLM KVConnectorBase_V1 implementation
  store.py       # Telys-backed block store
  codec.py       # Raw-byte / stacked-tensor codecs
  gpu_ops.py     # GPU injection helpers
  local_client.py # Slab / BlobStore client adapters
  telemetry.py   # JSONL instrumentation used by benches
tests/           # Unit and integration tests
bench/           # Benchmarks, run scripts, and RESULTS.md

Contributing

See CONTRIBUTING.md. The Telys engine runtime itself is closed and distributed separately; contributions here are limited to the open connector, tests, benchmarks, and docs.

License

Apache-2.0. See LICENSE and NOTICE.

Metadata

Release files for telys-vllm 0.1.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for telys-vllm 0.1.5
File Interpreter ABI Platform Actions
telys_vllm-0.1.5-py3-none-any.whl Python 3 none any Details for telys_vllm-0.1.5-py3-none-any.whl

Release files / telys_vllm-0.1.5-py3-none-any.whl

Download URL telys_vllm-0.1.5-py3-none-any.whl
Size 58.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
7893f618f08f9e7357d5fdf744a4bc3538218e27ff217bb8ffd830723cf265de
BLAKE2b-256 checksum
How to use checksums
970761d7d6d216debbdc3cf20f8350d0975b731513df2f279432fb70e1d8d67a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for telys-vllm 0.1.5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page