Skip to main content

cogcache

Semantic LLM answer cache — reuse paraphrased queries, cut latency and token cost.

PyPI Python License: MIT

cogcache caches LLM responses by semantic similarity instead of exact key match. When a paraphrased query arrives, it returns the previous answer in milliseconds — zero LLM tokens spent.

"What is semantic caching?"      → LLM call (4.2s, 320 tokens)
"What does semantic caching mean?" → Cache HIT (0.5ms, 0 tokens)   ← 99% savings

Install

pip install cogcache                   # core library
pip install cogcache[redis]            # + Redis Stack backend (HNSW vector search)
pip install cogcache[prometheus]       # + Prometheus metrics sink
pip install cogcache[openai-judge]     # + LLM-as-Judge quality scoring
pip install cogcache[langchain]        # + LangChain BaseCache adapter
pip install cogcache[all]              # everything

Quick start

from cogcache import CogniCache

cache = CogniCache(similarity_threshold=0.92)

def my_llm(query: str) -> str:
    # Your real LLM call here (OpenAI, Anthropic, DashScope, ...)
    return openai_client.chat.completions.create(...).choices[0].message.content

# First call → LLM
answer = cache.query("What is gradient descent?", llm_fn=my_llm)

# Second call → cache hit, zero LLM cost
answer = cache.query("Explain gradient descent.", llm_fn=my_llm)

As a decorator

@cache.cached(threshold=0.90)
def ask_llm(query: str) -> str:
    return my_llm(query)

ask_llm("What is X?")   # LLM call
ask_llm("Tell me X.")    # cache hit

With LangChain

from cogcache.integrations.langchain import CogniCacheLangChain
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(cache=CogniCacheLangChain(cache))

Features

Feature Default
Cosine-similarity semantic matching ✅
Pluggable stores: MemoryStore / RedisStore (Redis Stack HNSW) ✅
TTL eviction on read & write paths ✅
LLM-as-Judge with "write strict, hit lenient" policy optional
Prometheus + JSON metrics sink optional
Route / intent isolation (multi-tenant safe) ✅
Fail-open on backend failures ✅

Configuration

CogniCache(
    redis_url="redis://localhost:6379/0",   # None = in-memory
    similarity_threshold=0.92,               # 0.85–0.95 typical
    max_cache_size=10_000,
    ttl=3600,                                # -1 for no expiry
    vector_dim=512,                          # match your embedder
    enable_judge=True,                       # LLM Judge quality gate
    write_min_quality=0.8,
    judge_on_hit=False,                      # async hit-time warning
    embed_fn=my_custom_embedder,             # or use the default
    metrics=MetricsCollector(),              # observability hook
)

See tuning guide for threshold selection, embedding model comparison, and Prometheus alert thresholds.

When to use cogcache

✅ High-QPS chatbots where users phrase the same question different ways ✅ RAG systems with repetitive paraphrased queries
✅ Multi-tenant LLM APIs where you bill per token
✅ Demo / dev environments where you want to skip LLM calls on repeat

❌ Personalized answers (use route=user_id isolation if you must)
❌ Real-time data (weather, prices) — set short TTL or skip caching

Production readiness

  • ✅ Thread-safe MemoryStore and MetricsCollector
  • ✅ Fail-open: Redis disconnect / Judge crash never breaks your request path
  • ✅ 49 unit tests, run with pytest -q
  • ✅ Used in production at AI_Cost_Optimization reference deployment

Try it live

For a complete demo with FastAPI backend, admin dashboard, Prometheus exporter, and Docker Compose setup, see the cogcache-playground repo:

git clone https://github.com/AaronharveyHan/cogcache-playground.git
cd cogcache-playground && docker compose up
# Open http://localhost:8000/admin

License

MIT — see LICENSE.

Metadata

Release files for cogcache 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for cogcache 0.2.0
File Size Uploaded Provenance Actions
cogcache-0.2.0.tar.gz 41.4 kB With provenance Details for cogcache-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for cogcache 0.2.0
File Interpreter ABI Platform Actions
cogcache-0.2.0-py3-none-any.whl Python 3 none any Details for cogcache-0.2.0-py3-none-any.whl

Total release size: 76.4 kB

Release files / cogcache-0.2.0.tar.gz

Download URL cogcache-0.2.0.tar.gz
Size 41.4 kB
Tags Source
SHA-256 checksum
How to use checksums
8b6c15be13f3312681db2e70206a26e01154b1067e7f9638af897841d50cb81e
BLAKE2b-256 checksum
How to use checksums
114e7d9466acf8564ab1cc8743bf2c8812021484511348a123698debea3f1c5d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on May 23, 2026.

Transparency log

Release files / cogcache-0.2.0-py3-none-any.whl

Download URL cogcache-0.2.0-py3-none-any.whl
Size 35.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
3c0d4e5796969c0fbd4e506bc5ee250e6805e4ac85d0e6113826b7a62165b5b6
BLAKE2b-256 checksum
How to use checksums
9d84f5d8c1d9b59c68c72d1309b9543ebb666c153aae763077ed8b7feda583e8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on May 23, 2026.

Transparency log

Security for cogcache 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for cogcache-0.2.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on May 23, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page