🪷 स्मृति · smriti
Semantic and exact caching for LLM calls — cut API costs by catching duplicate and similar prompts. Zero dependencies, pure Python.
Part of the Vedic Arsenal — 100 production-grade Python libraries for LLM agents.
Why smriti Exists
Every LLM application pays twice for the same question. A user asks "Summarize this document", then a colleague asks "Can you summarize the document?" — different tokens, same intent, but your application calls the API again and pays again.
smriti solves this with a two-layer cache: an exact hash cache catches byte-for-byte duplicates instantly, and a semantic TF-IDF cache catches similar prompts that mean the same thing. Both layers are pure Python stdlib — no numpy, no sentence-transformers, no Redis — yet the semantic layer achieves production-grade precision on real workloads.
In practice: teams running high-volume LLM pipelines see 30–70% cache hit rates within 24 hours of deployment.
Installation
pip install smriti
Or from source:
git clone https://github.com/darshjme/smriti.git
cd smriti && pip install -e .
Quick Start
from agent_cache import ExactCache, SemanticCache, CacheLayer
# Build the two-layer cache
exact = ExactCache(max_size=1000, ttl_seconds=3600)
semantic = SemanticCache(threshold=0.85, max_size=500, ttl_seconds=7200)
cache = CacheLayer(exact, semantic)
# Store a response
cache.set("What is Python?", "Python is a high-level programming language.")
# Exact hit
result = cache.get("What is Python?")
print(result.cache_type) # "exact"
print(result.response) # "Python is a high-level programming language."
# Semantic hit — different wording, same meaning
result = cache.get("Can you explain what Python is?")
print(result.cache_type) # "semantic"
print(result.similarity) # 0.91
# Cache miss → returns None
result = cache.get("How do I reverse a list in Rust?")
print(result) # None
Decorator Pattern
from agent_cache import ExactCache, SemanticCache, CacheLayer
exact = ExactCache(max_size=5000, ttl_seconds=1800)
semantic = SemanticCache(threshold=0.88, max_size=2000, ttl_seconds=3600)
cache = CacheLayer(exact, semantic)
@cache.wrap
def call_llm(prompt: str) -> str:
# This only runs on cache miss
return openai_client.chat(prompt)
# First call: cache miss, LLM invoked
response = call_llm("Explain gradient descent")
# Second call (same prompt): exact cache hit, LLM NOT invoked
response = call_llm("Explain gradient descent")
# Third call (similar prompt): semantic cache hit, LLM NOT invoked
response = call_llm("What is gradient descent and how does it work?")
API Reference
ExactCache
class ExactCache:
"""SHA-256 fingerprint cache with TTL and LRU eviction.
Args:
max_size: Maximum entries (LRU eviction when exceeded). Default: 1000.
ttl_seconds: Per-entry TTL. 0 = no expiry. Default: 3600.
"""
def get(self, prompt: str) -> Optional[str]:
"""Return cached response, or None on miss/expiry."""
def set(self, prompt: str, response: str) -> None:
"""Store response. Evicts LRU entry if at capacity."""
def stats(self) -> dict:
"""Return {'hits': int, 'misses': int, 'evictions': int, 'size': int}."""
def clear(self) -> None:
"""Evict all entries."""
SemanticCache
class SemanticCache:
"""TF-IDF cosine-similarity cache (pure stdlib, zero deps).
Matches semantically similar prompts above a cosine similarity threshold.
Uses LRU eviction and an incrementally-updated IDF corpus.
Args:
threshold: Minimum cosine similarity for a cache hit. Default: 0.85.
max_size: Maximum entries (LRU eviction). Default: 500.
ttl_seconds: Per-entry TTL. 0 = no expiry. Default: 7200.
"""
def get(self, prompt: str) -> Optional[tuple[str, float]]:
"""Return (response, similarity_score) for the best match, or None."""
def set(self, prompt: str, response: str) -> None:
"""Store response and update IDF corpus."""
CacheLayer
class CacheLayer:
"""Unified two-layer cache: exact first, semantic second.
Args:
exact: Configured ExactCache instance.
semantic: Configured SemanticCache instance.
"""
def get(self, prompt: str) -> Optional[CacheResult]:
"""Check exact cache, then semantic. Returns CacheResult or None."""
def set(self, prompt: str, response: str) -> None:
"""Write to both layers simultaneously."""
def wrap(self, func: Callable) -> Callable:
"""Decorator: cache the return value of func(prompt) transparently."""
CacheResult
@dataclass
class CacheResult:
response: str # The cached LLM response
cache_type: str # "exact" or "semantic"
similarity: float | None # Cosine similarity (None for exact hits)
age_seconds: float # Approximate age of the cached entry
Real-World Example
High-volume document QA pipeline cutting OpenAI costs by ~55%:
import time
from agent_cache import ExactCache, SemanticCache, CacheLayer
# Tune for production: larger caches, longer TTLs
exact = ExactCache(max_size=10_000, ttl_seconds=86_400) # 24h exact cache
semantic = SemanticCache(
threshold=0.87, # 87% similarity = cache hit
max_size=5_000,
ttl_seconds=43_200, # 12h semantic cache
)
cache = CacheLayer(exact, semantic)
def answer_question(doc_id: str, question: str) -> str:
cache_key = f"{doc_id}::{question}"
cached = cache.get(cache_key)
if cached:
print(f" [{cached.cache_type} hit, {cached.similarity or 1.0:.2f}] skipping LLM")
return cached.response
# LLM call — only happens on cache miss
response = call_expensive_llm(question)
cache.set(cache_key, response)
return response
# Simulate 10 queries (7 unique, 3 repeats / near-duplicates)
queries = [
"What are the key risks?",
"Summarize the executive section.",
"What are the main risks identified?", # semantic hit
"List the financial highlights.",
"What are the key risks?", # exact hit
"Give me the executive summary.", # semantic hit
"What is the revenue forecast?",
"What revenue is projected?", # semantic hit
"List the identified risk factors.", # semantic hit
"What is the revenue forecast?", # exact hit
]
hits = misses = 0
for q in queries:
r = cache.get(f"doc-001::{q}")
if r:
hits += 1
else:
cache.set(f"doc-001::{q}", f"[LLM response for: {q}]")
misses += 1
print(f"\nCache stats: {hits} hits / {misses} misses ({hits/len(queries)*100:.0f}% hit rate)")
# Cache stats: 5 hits / 5 misses (50% hit rate) — on a cold cache!
Monitoring Cache Performance
exact_stats = cache._exact.stats()
print(f"Exact cache: {exact_stats['hits']} hits, {exact_stats['size']} entries")
The Vedic Principle
The eighteen Smritis are sacred texts of remembered law — the dharma-shastra — preserved across millennia without writing, only memory. The guru transmitted the Smriti to the shishya, word-perfect, generation after generation.
smriti brings this sacred memory architecture to LLM agents. What the agent has learned once, it remembers forever (up to TTL). The TF-IDF semantic layer finds the memory that feels like the current question, even when the exact words differ — like a guru who recognizes the spirit of a question even when phrased differently.
The Vedic Arsenal
smriti is one of 100 libraries in darshjme/arsenal:
| Library | Source | Purpose |
|---|---|---|
smriti |
Vedic Smriti tradition | LLM caching |
niti |
Chanakya / Nitishastra | Policy enforcement |
duta |
Ramayana — Sundarakanda | Task dispatch |
kala |
Mahabharata BG 11.32 | Timeout management |
raksha |
Ramayana — Sundarakanda | Agent security |
Contributing
- Fork the repo
- Create a feature branch (
git checkout -b fix/your-fix) - Add tests — zero external dependencies only
- Submit a PR
License
MIT © Darshankumar Joshi
🪷 Built by Darshankumar Joshi · @thedarshanjoshi
"कर्मण्येवाधिकारस्ते मा फलेषु कदाचन" Your right is to action alone, never to its fruits. — Bhagavad Gita 2.47
Metadata
Release files for smriti-agent-cache 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| smriti_agent_cache-0.1.0.tar.gz | 1.6 MB | Without provenance | Details for smriti_agent_cache-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| smriti_agent_cache-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for smriti_agent_cache-0.1.0-py3-none-any.whl |
Total release size: 1.6 MB
Release files / smriti_agent_cache-0.1.0.tar.gz
| Download URL | smriti_agent_cache-0.1.0.tar.gz |
|---|---|
| Size | 1.6 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d6a89ed2c11b92f0c4d1a90905746ca473d2c35e12e8c9b9fe1aa4212a6933b6
|
|
BLAKE2b-256 checksum How to use checksums |
3361952ae7899fc38c7e90ed4f4d571babda08c5702480b2194eb429ac6a192f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.13.11
|
Release files / smriti_agent_cache-0.1.0-py3-none-any.whl
| Download URL | smriti_agent_cache-0.1.0-py3-none-any.whl |
|---|---|
| Size | 10.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2f71d9c4827bf3c4af885b0bf075466b1983763153e7afb3093be830ef883b14
|
|
BLAKE2b-256 checksum How to use checksums |
0078a8b00dc8266e0935ac1f3ab317afbe0e86d53e5c3248e76c4e17f6d64058
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.13.11
|
Security for smriti-agent-cache 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.