semantic-cache-lib

Semantic caching for LLM calls via vector similarity search.

Instead of exact-match caching, semantic-cache-lib embeds your queries and finds semantically similar past answers — so "What is the capital of France?" and "Tell me the capital city of France" both hit the same cache entry.

Works as a decorator on any function that calls an LLM. Fully configurable: choose your embedder, backend, similarity threshold, and TTL.

Installation

# Core (in-memory backend only)
pip install semantic-cache-lib

# With specific extras
pip install semantic-cache-lib[openai]        # OpenAI embedder
pip install semantic-cache-lib[huggingface]   # HuggingFace Inference API embedder
pip install semantic-cache-lib[transformers]  # Local SentenceTransformers
pip install semantic-cache-lib[redis]         # Redis backend
pip install semantic-cache-lib[faiss]         # FAISS backend
pip install semantic-cache-lib[chroma]        # ChromaDB backend
pip install semantic-cache-lib[all]           # Everything

Quick Start

from semantic_cache import SemanticCache
from semantic_cache.embedders import HuggingFaceEmbedder

cache = SemanticCache(
    embedder=HuggingFaceEmbedder(
        token="hf_...",                              # or set HF_TOKEN env var
        model="sentence-transformers/all-MiniLM-L6-v2",
    ),
    threshold=0.90,   # 0.0–1.0 cosine similarity cutoff
    ttl=3600,         # cache entries expire after 1 hour (optional)
)

@cache
def call_llm(prompt: str) -> str:
    # your actual LLM call here
    return openai_client.chat.completions.create(...).choices[0].message.content

# First call → hits the LLM
response = call_llm("What is the capital of France?")

# Second call (semantically similar) → served from cache, LLM never called
response = call_llm("Tell me the capital city of France")

Async support

@cache
async def call_llm_async(prompt: str) -> str:
    return await async_openai_client.chat.completions.create(...)

response = await call_llm_async("What is the capital of France?")

Programmatic API

result = cache.get_or_set("my query", lambda: llm("my query"))
result = await cache.aget_or_set("my query", async_llm_call)

Embedders

HuggingFace Inference API

No GPU needed — calls the remote HF API.

from semantic_cache.embedders import HuggingFaceEmbedder

embedder = HuggingFaceEmbedder(
    token="hf_...",        # or HF_TOKEN env var
    model="sentence-transformers/all-MiniLM-L6-v2",
)

Local SentenceTransformers

Runs fully on-device. Downloads model from HF Hub on first use.

from semantic_cache.embedders import SentenceTransformerEmbedder

embedder = SentenceTransformerEmbedder(
    model="all-MiniLM-L6-v2",   # any sentence-transformers model
    device="cpu",                 # or "cuda"
)

OpenAI

from semantic_cache.embedders import OpenAIEmbedder

embedder = OpenAIEmbedder(
    api_key="sk-...",             # or OPENAI_API_KEY env var
    model="text-embedding-3-small",
)

Backends

In-Memory (default)

from semantic_cache.backends import MemoryBackend

backend = MemoryBackend(
    save_path="./cache.pkl",   # optional: persist across restarts
)

Redis

Requires Redis Stack for vector search.

from semantic_cache.backends import RedisBackend

backend = RedisBackend(
    url="redis://localhost:6379",
    vector_dim=384,              # must match your embedder's output dimension
)

FAISS

from semantic_cache.backends import FAISSBackend

backend = FAISSBackend(
    vector_dim=384,
    save_path="./faiss_cache",   # optional: persist across restarts
)

ChromaDB

from semantic_cache.backends import ChromaBackend

backend = ChromaBackend(
    url="http://localhost:8000",   # omit for in-process ephemeral client
    collection="llm_cache",
)

Full Configuration

from semantic_cache import SemanticCache
from semantic_cache.embedders import OpenAIEmbedder
from semantic_cache.backends import RedisBackend

cache = SemanticCache(
    embedder=OpenAIEmbedder(api_key="sk-..."),
    backend=RedisBackend(url="redis://localhost:6379", vector_dim=1536),
    threshold=0.92,                # similarity cutoff (default: 0.85)
    ttl=86400,                     # 24h TTL (default: None = no expiry)
    top_k=1,                       # candidates to retrieve (default: 1)
    on_hit=lambda q, r, s: print(f"HIT  sim={s:.3f} | {q[:60]}"),
    on_miss=lambda q: print(f"MISS | {q[:60]}"),
)

Observability

stats = cache.stats()
# CacheStats(hits=42, misses=8, total=50, hit_rate=84.00%, avg_similarity=0.9541)

cache.reset_stats()
cache.clear()          # remove all cache entries

Logging

Enable structured logging at any level:

import logging
logging.getLogger("semantic_cache").setLevel(logging.DEBUG)

Bring Your Own Embedder / Backend

from semantic_cache.embedders import AbstractEmbedder
from semantic_cache.backends import AbstractBackend

class MyEmbedder(AbstractEmbedder):
    def embed(self, text: str) -> list[float]:
        return my_model.encode(text)

class MyBackend(AbstractBackend):
    def store(self, key, vector, response, ttl=None): ...
    def search(self, vector, top_k=1) -> list[tuple[str, float]]: ...
    def delete(self, key): ...
    def clear(self): ...

Publishing / PyPI

# Install build tools
pip install hatch

# Build
hatch build

# Publish (or push a git tag to trigger GitHub Actions)
hatch publish

License

MIT — see LICENSE.