lmcache logo

A KV Cache Management Layer for Scalable LLM Inference

Blog | Documentation | Join Slack | Community Meeting | Roadmap

GitHub Repo stars PyPI PyPI - Downloads GitHub commit activity GitHub contributors Good first issues Slack Ask DeepWiki

⭐ If LMCache helps you serve LLMs faster and cheaper, give us a star — it helps more teams discover the project.

Updates

  • [2026/05] 🔥 Agentic workload benchmark on AMD MI300X (blog).
  • [2026/04] 🔥 LMCache's new multiprocess (MP) architecture release (blog).
  • [2026/03] LMCache at GTC 2026 (post).
  • [2026/01] LMCache multi-node P2P CPU memory sharing, from experimental feature to production (blog).
More
  • [2025/11] LMCache x CoreWeave accelerate efficient LLM inference for Cohere (blog).
  • [2025/10] LMCache joins the PyTorch Foundation and Tensormesh unveiled (blog, PyTorch).
  • [2025/09] NVIDIA Dynamo integrates LMCache, accelerating LLM inference (blog).
  • [2025/08] 🎉 LMCache hits 5,000+ GitHub stars (blog).
  • [2025/08] LMCache supports gpt-oss (20B/120B) on day 1 (blog).
  • [2025/07] Get faster LLM inference and cheaper responses with LMCache and Redis (Redis blog).
  • [2025/07] LMCache extends its turbo-boost to multimodal models in vLLM V1 (blog).
  • [2025/06] LLM Production Stack goes cross-hardware: AMD, Arm and Ascend (blog).

About

LMCache is a KV cache management layer for LLM inference. It turns KV cache from a temporary state into reusable AI-native knowledge that can be stored persistently, reused across multiple serving engines, monitored with an observability stack, and transformed for better generation quality. As a result, LMCache reduces TTFT (time-to-first-token) and improves throughput, especially for long-context agentic, multi-turn conversation, and knowledge-augmented workloads (e.g., RAG).

LMCache is vendor-neutral. It can be used as a KV cache layer for a range of mainstream open-source serving engines, inference frameworks, hardware vendors, storage systems, and infrastructure providers. The vendor neutrality allows users to freely switch between serving engines and storage vendors, while reusing the stored KV caches.

LMCache Deployment Modes

Key features

  • Engine-independent deployment: LMCache, as a standalone daemon process, manages KV cache independently from the inference engine process, so that KV cache will not be lost even if the inference engine crashes (i.e., no fate-sharing with engines).

  • Persistent, tiered KV cache offloading and reuse: Move KV caches out of GPU memory into a tiered storage hierarchy spanning CPU memory, local storage, and remote backends, enabling reuse across requests, sessions, and engine instances to reduce repeated prefill computation and improve TTFT.

  • Production-level KV cache observability: LMCache provides a rich set of KV cache observability metrics, including typical Kubernetes metrics (health monitoring, performance diagnostics), KV-cache-specific metrics (request-level and token-level prefix cache hits, lifecycle, request-level KV cache performance), management metrics (user-specific usage), and more.

  • Pluggable storage and transport backends: Easily integrate remote storage and KV transfer backends through a unified interface, enabling KV cache offloading and sharing across storage providers. Through this interface, LMCache supports storage backends including CPU RAM, local disk (SSD), Redis/Valkey, Mooncake, InfiniStore, S3-compatible object storage, NIXL, and GDS.

  • Non-prefix KV reuse: Extend KV reuse beyond prefix caching by reusing cached KV blocks at any position in the prompt. This leverages CacheBlend to selectively recompute tokens for quality recovery.

  • PD disaggregation and KV transfer: Support KV cache transfer from prefill workers to decode workers over NVLink, RDMA, or TCP through transport layers such as NIXL.

  • Pluggable KV transformation: A simple interface for researchers to write compression, token dropping, and custom serialization through a flexible SERDE interface.

LMCache is becoming an integral layer in the LLM inference ecosystem, with community-driven integration with serving engines, inference frameworks, hardware vendors, storage systems, and infrastructure providers:

LMCache ecosystem

Getting Started

To use LMCache, simply install lmcache from your package manager, e.g. pip:

pip install lmcache

For more setup options and examples, see:

Contributing

We welcome and value contributions and collaborations. Join us in improving LMCache. Check out the Contributing Guide or join our Slack community to get started.

Adoption and Partnerships

LMCache has a growing community of developers, researchers, industry adopters, and partners building the next generation of efficient LLM inference systems.

LMCache Adoption and Partnerships

As an independent open-source project, LMCache is becoming the de-facto standard for KV Cache management in LLM inference. Its continued development and community work are supported in part by Tensormesh.

Citation

LMCache builds on research in KV cache management, including cache reuse, offloading, compression, and serving optimization. If you use LMCache in your research, please cite the LMCache paper and related work.

@article{cheng2025lmcache,
  title={LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference},
  author={Cheng, Yihua and Liu, Yuhan and Yao, Jiayi and An, Yuwei and Chen, Xiaokun and Feng, Shaoting and Huang, Yuyang and Shen, Samuel and Du, Kuntai and Jiang, Junchen},
  journal={arXiv preprint arXiv:2510.09665},
  year={2025}
}
Related papers 
@inproceedings{liu2024cachegen,
  title={Cachegen: Kv cache compression and streaming for fast large language model serving},
  author={Liu, Yuhan and Li, Hanchen and Cheng, Yihua and Ray, Siddhant and Huang, Yuyang and Zhang, Qizheng and Du, Kuntai and Yao, Jiayi and Lu, Shan and Ananthanarayanan, Ganesh and others},
  booktitle={Proceedings of the ACM SIGCOMM 2024 Conference},
  pages={38--56},
  year={2024}
}

@inproceedings{yao2025cacheblend,
  title={Cacheblend: Fast large language model serving for rag with cached knowledge fusion},
  author={Yao, Jiayi and Li, Hanchen and Liu, Yuhan and Ray, Siddhant and Cheng, Yihua and Zhang, Qizheng and Du, Kuntai and Lu, Shan and Jiang, Junchen},
  booktitle={Proceedings of the twentieth European conference on computer systems},
  pages={94--109},
  year={2025}
}

License

The LMCache codebase is licensed under Apache License 2.0. See the LICENSE file for details.

Project links

Data verified by PyPI on Aug 20, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

3 maintainers

PyPI data
Data sourced directly from PyPI's database.
Avatar for apostaB from gravatar.com apostaB Avatar for deng451e from gravatar.com deng451e Avatar for LMCache from gravatar.com LMCache

Credits

Author: LMCache Team

GitHub Statistics

Data verified by PyPI on Aug 20, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

Apache-2.0
View SPDX License List

Requires

Python <3.14, >=3.10

Provides Extra

nixl

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lmcache-0.5.4.tar.gz (8.3 MB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

lmcache-0.5.4-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (16.4 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.27+ x86-64manylinux: glibc 2.28+ x86-64

lmcache-0.5.4-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl (16.2 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.26+ ARM64manylinux: glibc 2.28+ ARM64

lmcache-0.5.4-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (16.4 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.27+ x86-64manylinux: glibc 2.28+ x86-64

lmcache-0.5.4-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl (16.2 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.26+ ARM64manylinux: glibc 2.28+ ARM64

lmcache-0.5.4-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (16.3 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.27+ x86-64manylinux: glibc 2.28+ x86-64

lmcache-0.5.4-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl (16.2 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.26+ ARM64manylinux: glibc 2.28+ ARM64

lmcache-0.5.4-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (16.3 MB view details)

Uploaded CPython 3.10manylinux: glibc 2.27+ x86-64manylinux: glibc 2.28+ x86-64

lmcache-0.5.4-cp310-cp310-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl (16.1 MB view details)

Uploaded CPython 3.10manylinux: glibc 2.26+ ARM64manylinux: glibc 2.28+ ARM64

File details

Details for the file lmcache-0.5.4.tar.gz.

File metadata

  • Download URL: lmcache-0.5.4.tar.gz
  • Upload date:
  • Size: 8.3 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.7

File hashes

Hashes for lmcache-0.5.4.tar.gz
Algorithm Hash digest
SHA256 4e25826c76af65d537d253f1e4653fcbc36c34133bd2562c0d11ca19d20a692e
MD5 9578ccc4b4b53027c0e77b5e6247f29f
BLAKE2b-256 7d14831ea88428f276e47cb8cb267a4cc0b15add4166a7a91ff68349073d7a41

See more details on using hashes here.

Provenance

The following attestation bundles were made for lmcache-0.5.4.tar.gz:

Publisher: publish.yml on LMCache/LMCache

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lmcache-0.5.4-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for lmcache-0.5.4-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 bc7f11e06e4a5a015198c2bf4645d5ab67472776297b2abeace5a5829811680c
MD5 34e66968e0f8b05f8e7d35ad8f7f9ad0
BLAKE2b-256 52340d3b7f2b9d2106a78bfd28af97c5fa5b8ee29838817809a4ae77219ed987

See more details on using hashes here.

Provenance

The following attestation bundles were made for lmcache-0.5.4-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl:

Publisher: publish.yml on LMCache/LMCache

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lmcache-0.5.4-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for lmcache-0.5.4-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 0e13a3c4addc39b7de689144bc4dd1e1064bf0128bba6b7dcbfa3c2e6a0f9a2c
MD5 b3144293ed8e42125acff93b90dd36e3
BLAKE2b-256 03e4dde0dfbb9833facb814a436238f1bdb510e733aaf6fac177c9b704b00c77

See more details on using hashes here.

Provenance

The following attestation bundles were made for lmcache-0.5.4-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl:

Publisher: publish.yml on LMCache/LMCache

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lmcache-0.5.4-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for lmcache-0.5.4-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 d4fccafe6a0c90ffac96e3b67fd6dbe5a3a6ff8dbbe8139612a873c26971861b
MD5 d0d63f03654e2b3eba84780a11caf079
BLAKE2b-256 f67198b2af3a3b42551cf2c523bcd092097838e6dd71d0f166ddbdae920ffffd

See more details on using hashes here.

Provenance

The following attestation bundles were made for lmcache-0.5.4-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl:

Publisher: publish.yml on LMCache/LMCache

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lmcache-0.5.4-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for lmcache-0.5.4-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 a147fb8d704e0426366975130d835bc5f3560fe4c2dc167667bf57094c35897a
MD5 6eb87ea18c34e42f8b8d5aefd5d0eb45
BLAKE2b-256 a62e62d32be11074718ab5dd1d4798c63b8fad33c05824cebbc08020346b6ca5

See more details on using hashes here.

Provenance

The following attestation bundles were made for lmcache-0.5.4-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl:

Publisher: publish.yml on LMCache/LMCache

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lmcache-0.5.4-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for lmcache-0.5.4-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 39eb810fcbc096f10a288e551574e9be710662f5833efd9f42fa290500b9827b
MD5 c42811a11e2bec067632753012445b33
BLAKE2b-256 edb756c95a93ced2079384333b7cd6d2e6134cb1952f4e352955194739dbcf1f

See more details on using hashes here.

Provenance

The following attestation bundles were made for lmcache-0.5.4-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl:

Publisher: publish.yml on LMCache/LMCache

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lmcache-0.5.4-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for lmcache-0.5.4-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 024de58032f39ef296c5ecb5cc50aa7f9f0b50802e561346edfa91cc6de192d2
MD5 c8d6dedc7ce01e51241031a67c34736c
BLAKE2b-256 534d3438e085ff130f1afa53383e3afd4c121ccd8b475b0385d8461b6d7c352c

See more details on using hashes here.

Provenance

The following attestation bundles were made for lmcache-0.5.4-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl:

Publisher: publish.yml on LMCache/LMCache

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lmcache-0.5.4-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for lmcache-0.5.4-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 e97cb4ebcb09aa5a597e91f250b2ce5a7c33fc817424727add2842b6735efdae
MD5 4a4584c2f150c9ce28419b691b02a410
BLAKE2b-256 4fe4771b75924d414b7666f4c43f97b86a0a7bad5f298daceb95536b46b2ad9e

See more details on using hashes here.

Provenance

The following attestation bundles were made for lmcache-0.5.4-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl:

Publisher: publish.yml on LMCache/LMCache

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lmcache-0.5.4-cp310-cp310-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for lmcache-0.5.4-cp310-cp310-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 f50b55af350fb39f424083b20721c966dc071c7294c6917ce43b7bfba2622aff
MD5 fe0ef1a3fb63d1b826dd2a30fadc1290
BLAKE2b-256 367e93b1c19929ca0812d5584e03ca9c522cb16a1b2978e717972b065b51c8f3

See more details on using hashes here.

Provenance

The following attestation bundles were made for lmcache-0.5.4-cp310-cp310-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl:

Publisher: publish.yml on LMCache/LMCache

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page