temporal-cache-llm 0.1.0

Time-aware semantic LLM cache — automatically invalidates stale AI answers

temporal-cache

Time-aware semantic caching for LLM responses

Cache LLM responses by semantic meaning AND freshness. Stop serving stale answers.

---

Table of Contents

• What is temporal-cache?
• The Problem
• How It Works
• Installation
• Quick Start
• Python API Reference
• Staleness Classes
• Architecture
• Configuration
• Testing
• Contributing
• Roadmap
• License

---

What is temporal-cache?

temporal-cache is a semantic cache for LLM API responses that understands when an answer was relevant, not just what it says.

Traditional semantic caches store responses and return them when a similar query arrives. But they don't know if that response is still correct. A cached answer about "current US president" from 2024 is semantically relevant in 2026 — but wrong.

temporal-cache solves this with a StalenessClass classifier that evaluates how time-sensitive a query is, and an optional LLM-based semantic classifier that detects when cached content has drifted from current facts.

Use cases:

• Knowledge bases with evolving information
• Customer support bots with changing policies
• Financial/news Q&A systems
• Any LLM application where answers expire

---

The Problem

┌─────────────────────────────────────────────────────────────────┐
│  Traditional Semantic Cache                                      │
│                                                                  │
│  Query: "What is the capital of France?"                         │
│  Cache hit → "Paris" ✓                                           │
│                                                                  │
│  Query: "What's the current CEO of Tesla?"                       │
│  Cache hit → "Elon Musk" (cached 2024)                           │
│  Reality: CEO changed in 2025 ✗                                   │
│                                                                  │
│  Problem: Cache doesn't know the answer is stale                 │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│  temporal-cache                                                  │
│                                                                  │
│  Query: "What's the current CEO of Tesla?"                       │
│  StalenessClass: DYNAMIC (changes frequently)                    │
│  Action: Revalidate → check if cache is still valid              │
│  Result: Fresh answer or cache with metadata about freshness     │
└─────────────────────────────────────────────────────────────────┘

---

How It Works

Cache Flow

                          ┌──────────────┐
                          │   Incoming   │
                          │    Query     │
                          └──────┬───────┘
                                 │
                                 ▼
                    ┌────────────────────────┐
                    │  StalenessClass        │
                    │  Classifier            │
                    │                        │
                    │  STATIC │ EVOLVING │   │
                    │  SEMI-  │ DYNAMIC  │   │
                    │  STATIC │          │   │
                    └────────┬───────────┘
                             │
              ┌──────────────┼──────────────┐
              │              │              │
              ▼              ▼              ▼
         ┌─────────┐   ┌─────────┐   ┌─────────┐
         │ STATIC  │   │ SEMI-   │   │ DYNAMIC │
         │         │   │ STATIC  │   │         │
         │ Cache   │   │ Cache   │   │ Reval-  │
         │ Forever │   │ + TTL   │   │ idate   │
         └─────────┘   └─────────┘   └─────────┘
              │              │              │
              │              │              │
              ▼              ▼              ▼
         ┌─────────────────────────────────────────┐
         │           Semantic Similarity           │
         │              (embedding)                │
         └──────────────────┬──────────────────────┘
                            │
              ┌─────────────┴─────────────┐
              │                           │
              ▼                           ▼
         Hit (≥ threshold)          Miss
              │                           │
              ▼                           ▼
         ┌──────────┐              ┌──────────────┐
         │ Return   │              │ Call LLM     │
         │ Cached   │              │ Store with   │
         │ Response │              │ metadata     │
         └──────────┘              └──────────────┘

Staleness Classification

Each query is classified into one of four staleness classes:

┌──────────────────────────────────────────────────────────────────┐
│                                                                  │
│  Query Analysis                                                  │
│  ═══════════════                                                 │
│                                                                  │
│  "What is 2+2?"                    → STATIC                     │
│  "What is Python?"                 → STATIC                     │
│  "Explain quantum computing"       → SEMI_STATIC                │
│  "What are Python best practices?" → SEMI_STATIC                │
│  "Who is the current US president?"→ DYNAMIC                    │
│  "Tesla stock price today"         → DYNAMIC                    │
│  "Latest news about OpenAI"        → EVOLVING                   │
│  "What changed in Python 3.12?"    → EVOLVING                   │
│                                                                  │
└──────────────────────────────────────────────────────────────────┘

---

Installation

pip install temporal-cache

From source:

git clone https://github.com/yourusername/temporal-cache.git
cd temporal-cache
pip install -e .

With Redis support:

pip install temporal-cache[redis]

Requirements

• Python 3.9+
• sentence-transformers (for embeddings)
• redis (optional, for Redis backend)

---

Quick Start

from temporal_cache import TemporalCache

# Initialize with default settings (in-memory backend)
cache = TemporalCache()

# Or with Redis backend
cache = TemporalCache(
    backend="redis",
    redis_url="redis://localhost:6379",
    embedding_model="all-MiniLM-L6-v2"
)

# Store a response
cache.set(
    query="What is the capital of France?",
    response="Paris is the capital of France.",
    metadata={"source": "wikipedia", "verified": True}
)

# Retrieve a response (semantic match)
result = cache.get("What's the capital of France?")

if result:
    print(f"Cache hit: {result.response}")
    print(f"Staleness: {result.staleness_class}")
    print(f"Freshness: {result.freshness_score}")
else:
    print("Cache miss - call your LLM here")

---

Python API Reference

TemporalCache

TemporalCache(
    backend: str = "memory",           # "memory" or "redis"
    redis_url: str = "redis://localhost:6379",
    embedding_model: str = "all-MiniLM-L6-v2",
    similarity_threshold: float = 0.85,
    ttl_config: dict = None,           # Custom TTL per staleness class
    llm_classifier: bool = False,      # Enable LLM-based revalidation
    classifier_model: str = None       # Model for LLM classifier
)

Methods

set(query, response, metadata=None, staleness_class=None)

Store a query-response pair in the cache.

Parameter	Type	Description
query	str	The input query
response	str	The LLM response to cache
metadata	dict	Optional metadata (source, timestamp, etc.)
staleness_class	StalenessClass	Override auto-detected class

get(query, force_refresh=False)

Retrieve a cached response by semantic similarity.

Parameter	Type	Description
query	str	The query to look up
force_refresh	bool	Bypass cache, always return None

Returns CacheResult or None.

invalidate(query=None, pattern=None)

Remove entries from cache.

# Invalidate specific query
cache.invalidate(query="What is the capital of France?")

# Invalidate by pattern
cache.invalidate(pattern="*stock*")

stats()

Return cache statistics.

stats = cache.stats()
# {
#     "total_entries": 1523,
#     "hits": 892,
#     "misses": 231,
#     "stale_rejections": 45,
#     "avg_freshness_score": 0.87
# }

CacheResult

@dataclass
class CacheResult:
    response: str
    similarity_score: float
    staleness_class: StalenessClass
    freshness_score: float          # 0.0 (stale) to 1.0 (fresh)
    cached_at: datetime
    metadata: dict

StalenessClass

class StalenessClass(Enum):
    STATIC = "static"           # Never changes
    SEMI_STATIC = "semi_static" # Changes rarely
    EVOLVING = "evolving"       # Changes periodically
    DYNAMIC = "dynamic"         # Changes frequently

---

Staleness Classes

Class	Description	TTL	Revalidation	Examples
STATIC	Facts that don't change	∞	Never	"What is π?", "2+2", "What is DNA?"
SEMI_STATIC	Rarely changes	30 days	On major events	"Python docs", "Physics laws", "Best practices"
EVOLVING	Changes periodically	24 hours	On schedule	"Python 3.12 features", "Company policies"
DYNAMIC	Changes constantly	1 hour	Every query	"Stock price", "Current president", "Live news"

Custom TTL Configuration

cache = TemporalCache(
    ttl_config={
        "static": None,              # No expiration
        "semi_static": 86400 * 30,   # 30 days
        "evolving": 86400,           # 1 day
        "dynamic": 3600              # 1 hour
    }
)

---

Architecture

temporal_cache/
├── __init__.py
├── cache.py              # TemporalCache engine
├── classifiers/
│   ├── __init__.py
│   ├── staleness.py      # StalenessClass rule-based classifier
│   └── llm.py            # LLM-based semantic classifier
├── backends/
│   ├── __init__.py
│   ├── memory.py         # In-memory backend (dict)
│   └── redis.py          # Redis backend
├── embeddings.py         # Embedding generation
├── models.py             # CacheResult, CacheEntry dataclasses
├── config.py             # Configuration and defaults
└── utils.py              # Helpers

Data Flow

┌─────────────────────────────────────────────────────────────────────┐
│                                                                     │
│  User Query                                                         │
│       │                                                             │
│       ▼                                                             │
│  ┌─────────────┐     ┌──────────────────┐                           │
│  │ Staleness   │────▶│ Embedding        │                           │
│  │ Classifier  │     │ Generator        │                           │
│  └──────┬──────┘     └────────┬─────────┘                           │
│         │                     │                                     │
│         ▼                     ▼                                     │
│  ┌─────────────┐     ┌──────────────────┐                           │
│  │ TTL         │     │ Backend          │                           │
│  │ Manager     │     │ (Memory/Redis)   │                           │
│  └──────┬──────┘     └────────┬─────────┘                           │
│         │                     │                                     │
│         └──────────┬──────────┘                                     │
│                    │                                                │
│                    ▼                                                │
│         ┌──────────────────────┐                                    │
│         │ Response Validator   │                                    │
│         │                      │                                    │
│         │ - Check similarity   │                                    │
│         │ - Check freshness    │                                    │
│         │ - LLM revalidation   │                                    │
│         └──────────┬───────────┘                                    │
│                    │                                                │
│         ┌──────────┴──────────┐                                     │
│         │                     │                                     │
│         ▼                     ▼                                     │
│    Cache Hit              Cache Miss                                │
│    (return)               (call LLM, store)                         │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

---

Configuration

Environment Variables

TEMPORAL_CACHE_BACKEND=redis
TEMPORAL_CACHE_REDIS_URL=redis://localhost:6379
TEMPORAL_CACHE_SIMILARITY_THRESHOLD=0.85
TEMPORAL_CACHE_DEFAULT_TTL=3600

Configuration File

# temporal_cache.yaml
backend: redis
redis_url: redis://localhost:6379
embedding_model: all-MiniLM-L6-v2
similarity_threshold: 0.85
llm_classifier:
  enabled: true
  model: gpt-4
  temperature: 0.0
ttl:
  static: null
  semi_static: 2592000
  evolving: 86400
  dynamic: 3600

---

Testing

Run the test suite:

# All tests
pytest

# With coverage
pytest --cov=temporal_cache --cov-report=html

# Verbose output
pytest -v

The test suite includes 16 tests covering:

• Cache storage and retrieval
• Semantic similarity matching
• Staleness classification accuracy
• TTL expiration behavior
• Redis backend integration
• LLM classifier revalidation
• Edge cases and error handling

---

Contributing

Contributions are welcome! Here's how to get started:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Run the tests (pytest)
5. Submit a pull request

Development Setup

git clone https://github.com/yourusername/temporal-cache.git
cd temporal-cache
pip install -e ".[dev]"
pre-commit install

Code Style

• Follow PEP 8
• Use type hints
• Write docstrings for public APIs
• Add tests for new features

---

Roadmap

• v0.2.0 - Async support (async/await)
• v0.2.0 - PostgreSQL backend
• v0.3.0 - Bloom filter pre-check for fast misses
• v0.3.0 - Distributed cache invalidation
• v0.4.0 - Embedding cache (reuse embeddings across queries)
• v0.4.0 - Web dashboard for cache analytics
• v1.0.0 - Production-ready release

---

License

MIT License. See LICENSE for details.

---

Built with care for the LLM community