llmgatekeeper 0.2.0

Semantic caching library for LLM and RAG systems

LLMGatekeeper

A semantic caching library for LLM and RAG systems. Eliminates redundant LLM API calls and vector database queries by recognising semantically equivalent queries using embedding-based similarity matching.

Instead of requiring exact string matches, LLMGatekeeper understands that "What is Python?" and "Tell me about Python" are asking the same thing — and returns the cached response instantly.

Features

• Semantic matching — Embedding-based similarity search catches paraphrases, synonyms, and rewording automatically
• Redis backends — Simple hash-based mode for small datasets (<10k entries); auto-upgrades to RediSearch when available, using VECTOR_RANGE for threshold-filtered queries and KNN for unfiltered ones
• Pluggable embeddings — Ships with all-MiniLM-L6-v2 (Sentence-Transformers) out of the box; swap in OpenAI embeddings or any custom provider
• Confidence levels — Scores are classified into HIGH / MEDIUM / LOW / NONE bands with per-model tuned thresholds
• Multi-tenant isolation — Namespace prefixing keeps separate tenants' caches partitioned in the same Redis instance
• Full async stack — AsyncSemanticCache plus AsyncRedisBackend and AsyncRediSearchBackend; SentenceTransformerProvider.aembed offloads to a thread so encoding never blocks the event loop
• Analytics & observability — Hit/miss rates, latency percentiles (p50/p95/p99), near-miss tracking, and top-query frequency — implemented with a single backend call per get so analytics doesn't double request volume
• TTL control — Global default TTL with per-entry overrides; ttl=0 disables expiry for specific entries
• Bulk warming — Load many entries at once with warm(), including batch-size control and progress callbacks
• Safety rails — Backends record the embedding dimension on first write and reject mismatched vectors; the RediSearch backend also validates an existing index's DIM on attach so switching embedding models can't silently corrupt results

Installation

pip install llmgatekeeper

With OpenAI embedding support:

pip install llmgatekeeper[openai]

For development (tests, linting, type-checking):

pip install llmgatekeeper[dev]

Requirements

• Python 3.9+
• Redis 6.2+ (Redis Stack recommended for RediSearch vector search)

Quick Start

import redis
from llmgatekeeper import SemanticCache

# Connect to your Redis instance
client = redis.Redis(host="localhost", port=6379)

# Create a semantic cache (auto-detects RediSearch if available)
cache = SemanticCache(client)

# Store a response
cache.set("What is Python?", "Python is a high-level programming language.")

# Retrieve with a semantically similar query — no exact match needed
result = cache.get("Tell me about Python")
print(result)  # "Python is a high-level programming language."

With metadata

cache.set(
    "What is Python?",
    "Python is a high-level programming language.",
    metadata={"source": "docs", "version": "3.12"},
)

result = cache.get("Explain Python", include_metadata=True)
if result:
    print(result.response)   # the cached response
    print(result.similarity) # e.g. 0.91
    print(result.metadata)   # {"source": "docs", "version": "3.12"}

User metadata keys are preserved verbatim — including reserved-sounding names like query or response. The cache stores its own bookkeeping under a single namespaced key (_llmgk) so it never collides with what you pass in.

Async usage

import asyncio
import redis.asyncio as aioredis
from llmgatekeeper import AsyncSemanticCache
from llmgatekeeper.backends import create_async_redis_backend

async def main():
    client = aioredis.Redis(host="localhost", port=6379)

    # Auto-detects RediSearch on the async client and returns the right backend.
    backend = await create_async_redis_backend(client)
    cache = AsyncSemanticCache(backend)

    await cache.set("What is Python?", "A high-level programming language.")
    result = await cache.get("Tell me about Python")
    print(result)

asyncio.run(main())

For direct backend construction skip the factory:

from llmgatekeeper.backends import AsyncRedisBackend, AsyncRediSearchBackend

# Plain Redis (brute-force similarity, <10k entries)
backend = AsyncRedisBackend(client)

# Redis Stack with RediSearch
backend = AsyncRediSearchBackend(client, vector_dimension=384)
await backend.connect()   # verifies the module and creates/validates the index

Multi-tenant isolation

tenant_a = SemanticCache(client, namespace="tenant_a")
tenant_b = SemanticCache(client, namespace="tenant_b")

tenant_a.set("Hello", "Hi from A")
tenant_b.set("Hello", "Hi from B")

print(tenant_a.get("Hello").response)  # "Hi from A"
print(tenant_b.get("Hello").response)  # "Hi from B"

TTL (time-to-live)

# Global default: all entries expire after 1 hour
cache = SemanticCache(client, default_ttl=3600)

# Per-entry override: this entry lives forever
cache.set("Permanent fact", "Water is H2O", ttl=0)

# Short-lived entry: expires in 60 seconds
cache.set("Breaking news", "...", ttl=60)

Analytics

cache = SemanticCache(client, enable_analytics=True)

# ... use the cache normally ...

stats = cache.stats()
print(f"Hit rate:      {stats.hit_rate:.1%}")
print(f"P95 latency:   {stats.p95_latency_ms:.2f} ms")
print(f"Near misses:   {len(stats.near_misses)}")
print(f"Top queries:   {[(q.query, q.count) for q in stats.top_queries[:3]]}")

Analytics is opt-in via enable_analytics=True. Each get issues exactly one backend round trip whether it hits or misses — near-miss tracking reuses the same result.

Error handling

from llmgatekeeper.exceptions import (
    CacheError,             # base for everything below
    BackendError,           # Redis-side failures (raised on dim mismatch too)
    BackendConnectionError,
    BackendTimeoutError,
    EmbeddingError,         # embedding model / API failures
    ConfigurationError,     # invalid construction args
)

try:
    cache.set("q", "r")
except BackendError as e:
    print("Storage failed:", e, "(original:", e.original_error, ")")

The library never shadows Python builtins — BackendConnectionError and BackendTimeoutError are explicitly prefixed.

Full examples

The snippets above cover the basics. For a complete walkthrough of every feature — backends, embedding providers, similarity metrics, confidence classification, retrieval, async, analytics, logging, error handling, and custom backend patterns — see examples/quickstart_redis.py.

Architecture

The library is organised into three layers:

┌────────────────────────────────────────────────────────┐
│        SemanticCache / AsyncSemanticCache API          │
│        (get / set / delete / warm / stats …)           │
├─────────────────┬──────────────────────────────────────┤
│ Embedding       │ Similarity Engine                    │
│ Engine          │ (cosine, fixed at the backend)       │
│ (pluggable      │ (confidence classification)          │
│  providers)     │ (multi-result retrieval)             │
├─────────────────┴──────────────────────────────────────┤
│              Storage Backend Adapter                   │
│  RedisSimpleBackend       │ RediSearchBackend          │
│  AsyncRedisBackend        │ AsyncRediSearchBackend     │
│  (brute-force, pipelined) │ (VECTOR_RANGE + KNN)       │
└────────────────────────────────────────────────────────┘

See ARCHITECTURE.md for a detailed breakdown of every layer, class hierarchy, and extension point.

Configuration Reference

Parameter	Default	Description
threshold	0.85	Minimum cosine similarity for a cache hit
default_ttl	None	Seconds until entries expire (None = no expiry)
namespace	"default"	Key prefix for multi-tenant isolation
embedding_provider	SentenceTransformerProvider	Provider used to generate query embeddings
enable_analytics	False	Track hit/miss rate, latency percentiles, and near-misses

Cosine is the only similarity metric currently supported; the backend index is created with DISTANCE_METRIC=COSINE. SimilarityMetric.DOT_PRODUCT and SimilarityMetric.EUCLIDEAN exist as standalone helpers in llmgatekeeper.similarity.metrics but are not wired through SemanticCache.

Running Tests

pip install llmgatekeeper[dev]
pytest

The test suite contains 497 tests covering all backends (sync + async, simple + RediSearch), embedding providers, similarity metrics, async paths, error handling, analytics, dimension validation, and edge cases.

Contributing

Contributions are welcome. Please open an issue or pull request on GitHub.

License

This project is licensed under the MIT License — see the LICENSE file for details.