cogcache 0.2.0

Semantic LLM answer cache — reuse paraphrased queries, cut latency and token cost.

cogcache

Semantic LLM answer cache — reuse paraphrased queries, cut latency and token cost.

cogcache caches LLM responses by semantic similarity instead of exact key match. When a paraphrased query arrives, it returns the previous answer in milliseconds — zero LLM tokens spent.

"What is semantic caching?"      → LLM call (4.2s, 320 tokens)
"What does semantic caching mean?" → Cache HIT (0.5ms, 0 tokens)   ← 99% savings

Install

pip install cogcache                   # core library
pip install cogcache[redis]            # + Redis Stack backend (HNSW vector search)
pip install cogcache[prometheus]       # + Prometheus metrics sink
pip install cogcache[openai-judge]     # + LLM-as-Judge quality scoring
pip install cogcache[langchain]        # + LangChain BaseCache adapter
pip install cogcache[all]              # everything

Quick start

from cogcache import CogniCache

cache = CogniCache(similarity_threshold=0.92)

def my_llm(query: str) -> str:
    # Your real LLM call here (OpenAI, Anthropic, DashScope, ...)
    return openai_client.chat.completions.create(...).choices[0].message.content

# First call → LLM
answer = cache.query("What is gradient descent?", llm_fn=my_llm)

# Second call → cache hit, zero LLM cost
answer = cache.query("Explain gradient descent.", llm_fn=my_llm)

As a decorator

@cache.cached(threshold=0.90)
def ask_llm(query: str) -> str:
    return my_llm(query)

ask_llm("What is X?")   # LLM call
ask_llm("Tell me X.")    # cache hit

With LangChain

from cogcache.integrations.langchain import CogniCacheLangChain
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(cache=CogniCacheLangChain(cache))

Features

Feature	Default
Cosine-similarity semantic matching	✅
Pluggable stores: MemoryStore / RedisStore (Redis Stack HNSW)	✅
TTL eviction on read & write paths	✅
LLM-as-Judge with "write strict, hit lenient" policy	optional
Prometheus + JSON metrics sink	optional
Route / intent isolation (multi-tenant safe)	✅
Fail-open on backend failures	✅

Configuration

CogniCache(
    redis_url="redis://localhost:6379/0",   # None = in-memory
    similarity_threshold=0.92,               # 0.85–0.95 typical
    max_cache_size=10_000,
    ttl=3600,                                # -1 for no expiry
    vector_dim=512,                          # match your embedder
    enable_judge=True,                       # LLM Judge quality gate
    write_min_quality=0.8,
    judge_on_hit=False,                      # async hit-time warning
    embed_fn=my_custom_embedder,             # or use the default
    metrics=MetricsCollector(),              # observability hook
)

See tuning guide for threshold selection, embedding model comparison, and Prometheus alert thresholds.

When to use cogcache

✅ High-QPS chatbots where users phrase the same question different ways ✅ RAG systems with repetitive paraphrased queries
✅ Multi-tenant LLM APIs where you bill per token
✅ Demo / dev environments where you want to skip LLM calls on repeat

❌ Personalized answers (use route=user_id isolation if you must)
❌ Real-time data (weather, prices) — set short TTL or skip caching

Production readiness

• ✅ Thread-safe MemoryStore and MetricsCollector
• ✅ Fail-open: Redis disconnect / Judge crash never breaks your request path
• ✅ 49 unit tests, run with pytest -q
• ✅ Used in production at AI_Cost_Optimization reference deployment

Try it live

For a complete demo with FastAPI backend, admin dashboard, Prometheus exporter, and Docker Compose setup, see the cogcache-playground repo:

git clone https://github.com/AaronharveyHan/cogcache-playground.git
cd cogcache-playground && docker compose up
# Open http://localhost:8000/admin

License

MIT — see LICENSE.