llm-cache

SQLite-backed LLM response cache. Exact match + fuzzy match. Decorator API. Zero mandatory server dependencies.

Why llm-cache?

Every LLM API call costs money and takes time. In development and testing, you hit the same prompts over and over. In production, users ask the same questions. llm-cache stores responses in a local SQLite database and serves them instantly — no Redis, no server, no external service.

Two match layers:

• Exact match — SHA-256 hash of the normalised prompt. O(1) lookup.
• Fuzzy match — token overlap similarity for near-identical prompts. Catches rephrasings without embeddings.

Install

pip install llm-cache

Quick start

from llm_cache import LLMCache

cache = LLMCache()  # stores in .llm_cache.db

# Decorator — wraps any LLM call function
@cache.cached(model="claude-opus-4-5")
def ask(prompt: str) -> str:
    return client.messages.create(
        model="claude-opus-4-5",
        messages=[{"role": "user", "content": prompt}],
    ).content[0].text

ask("What is RAG?")   # hits API first time
ask("What is RAG?")   # served from cache instantly — no API call
ask("What is rag?")   # fuzzy match — also served from cache

Direct API

from llm_cache import LLMCache, MatchType

cache = LLMCache("myapp.db", fuzzy_threshold=0.85, ttl_seconds=86400)

# Get
response, match = cache.get("What is RAG?")
if match == MatchType.MISS:
    response = call_api("What is RAG?")
    cache.set("What is RAG?", response, model="claude-opus-4-5", tags=["qa"])

# Match types
# MatchType.EXACT  — identical prompt
# MatchType.FUZZY  — near-match above threshold
# MatchType.MISS   — not cached

# Stats
print(cache.stats())
# CacheStats(entries=142, hit_rate=73%, tokens_saved≈48,200)

# Management
cache.delete(tags=["qa"])              # delete by tag
cache.delete(older_than_seconds=3600)  # expire old entries
cache.delete(model="claude-opus-4-5") # delete by model
cache.clear()                          # wipe everything

Configuration

cache = LLMCache(
    db_path=".llm_cache.db",    # SQLite file path
    fuzzy_threshold=0.85,        # 0-1, how similar prompts must be for fuzzy hit
    ttl_seconds=86400,           # auto-expire entries after N seconds (None = never)
    max_entries=10000,           # evict oldest when cache exceeds this size
)

Cost savings estimate

stats = cache.stats()
print(f"Tokens saved:  {stats.estimated_tokens_saved:,}")
print(f"Estimated saving: ~£{stats.cost_savings_estimate:.2f}")
print(f"Hit rate: {stats.hit_rate:.0%}")

Context manager

with LLMCache("session.db") as cache:
    result, _ = cache.get("my prompt")

Linda Oraegbunam | LinkedIn | Twitter | GitHub