A semantic cache for LLM calls.
If a new question means the same as an old one, return the saved answer.
Track your hit rate and how much money it saved.
Why
Users ask the same things in different words. An exact-match cache misses
"How do I reverse a list in Python?" vs "python: how to reverse a list", so you pay
for the same answer twice. semcache embeds each prompt, finds the closest one already
answered, and returns that answer when the two are similar enough. Every hit is counted
in tokens and dollars, so you can see what the cache is worth.
- Drop-in: wrap an
OpenAIorAnthropicclient, decorate a function, or callget/setyourself. - Shows savings: hit rate, exact vs semantic hits, tokens saved, money saved and spent, broken down per model.
- Safe defaults: answers never cross models, namespaces, system prompts or conversation history. Truncated or refused responses are not cached.
- Small: the core needs only
numpy. Neural embedders are optional extras. - Persistent: one SQLite file holds entries and stats. TTL and LRU size limits are built in.
Install
pip install semcacher # core (numpy only)
pip install "semcacher[local]" # + sentence-transformers, runs locally
pip install "semcacher[openai]" # + OpenAI embeddings
pip install "semcacher[redis]" # + shared cache in Redis
You install semcacher and import semcache:
from semcache import SemanticCache
From source:
pip install git+https://github.com/sjapanjots/Semantic_Cache_LLM
Quickstart
from semcache import SemanticCache, LLMResult
cache = SemanticCache(pricing={"demo-model": (2.50, 10.00)}) # USD per 1M input/output tokens
def call_llm(prompt):
... # your real LLM call
return LLMResult("answer", input_tokens=1200, output_tokens=400)
cache.get_or_call("How do I reverse a list in Python?", lambda: call_llm(...), model="demo-model")
cache.get_or_call("python: how to reverse a list", lambda: call_llm(...), model="demo-model") # cache hit
print(cache.stats)
Output of examples/quickstart.py, which needs no API key:
lookups 6
hits 3 (50.0%)
exact 1
semantic 2
misses 3
tokens saved 4,800
money saved $0.0210
money spent $0.0210
savings rate 50.0%
Usage
1. Wrap your client (no other code changes)
from anthropic import Anthropic
from semcache import SemanticCache, SentenceTransformerEmbedder
from semcache.integrations import wrap_anthropic
cache = SemanticCache(embedder=SentenceTransformerEmbedder(), path="cache.db")
client = wrap_anthropic(Anthropic(), cache)
client.messages.create(
model="claude-opus-5-5",
max_tokens=1024,
messages=[{"role": "user", "content": "How should a Python dev learn Rust?"}],
)
wrap_openai(OpenAI(), cache) does the same for client.chat.completions.create.
Async clients (AsyncAnthropic, AsyncOpenAI) work too. On a hit you get back the
SDK's normal response object, rebuilt from the cache.
How requests are matched:
| Part of the request | Treatment |
|---|---|
| Last user message | Compared by meaning |
| Model | Must match exactly |
| System prompt, earlier turns, tools, temperature, max tokens, ... | Hashed; must match exactly |
timeout, extra_headers, user, metadata |
Ignored |
stream=True |
Passed through, not cached |
Responses stopped by max_tokens / refusal / length / content_filter |
Not cached |
2. Decorate a function
@cache.cached(model="claude-sonnet-5-5")
def ask(prompt: str) -> LLMResult:
response = my_llm(prompt)
return LLMResult(response.text, input_tokens=response.in_tok, output_tokens=response.out_tok)
ask("How many days are in a leap year?")
ask("how many days does a leap year have") # served from the cache
Works on async def functions too. Use key=lambda *a, **kw: ... to choose which
argument is the prompt.
3. Call get / set yourself
answer = cache.get(prompt, model="claude-opus-5-5")
if answer is None:
answer, usage = call_llm(prompt)
cache.set(prompt, answer, model="claude-opus-5-5",
input_tokens=usage.input, output_tokens=usage.output)
cache.lookup(...) returns a CacheHit with .response, .similarity, .exact,
.cost_saved and the stored .entry.
Choosing an embedder and threshold
The threshold is the minimum cosine similarity for a semantic hit. Each embedder ships a conservative default; tune it on your own traffic.
| Embedder | Install | Default threshold | Notes |
|---|---|---|---|
HashingEmbedder (default) |
core | 0.70 | Lexical (words and character n-grams). No model download. Catches rewording, not synonyms. |
SentenceTransformerEmbedder() |
[local] |
0.85 | Real semantic matching, runs on CPU. all-MiniLM-L6-v2 by default. |
OpenAIEmbedder() |
[openai] |
0.90 | text-embedding-3-small by default. Costs a small amount per lookup. |
CallableEmbedder(fn, name) |
core | 0.85 | Bring any model: Voyage, Cohere, an in-house service... |
Too many wrong answers? Raise the threshold. Too few hits? Lower it, or switch to a
neural embedder. You can also override it per call: cache.lookup(p, threshold=0.95).
Calibrating the threshold from your own data
Instead of guessing, label some prompt pairs as same or different meaning and let
calibrate pick the threshold:
from semcache import HashingEmbedder, SemanticCache, calibrate, load_pairs
pairs = load_pairs("examples/tuning_pairs.csv") # CSV columns: a,b,same
result = calibrate(HashingEmbedder(), pairs) # or target_precision=0.99
print(result.summary())
cache = SemanticCache(threshold=result.threshold)
embedder hashing-1024-3
pairs 20 (10 same, 10 different)
recommend threshold=0.7099 (best F0.5)
threshold precision recall wrong hits F0.5
0.7000 83.3% 50.0% 1 0.735
0.7083 83.3% 50.0% 1 0.735
0.7099 100.0% 50.0% 0 0.833 <-
0.7100 100.0% 40.0% 0 0.769
Precision is the share of hits that are correct; recall is the share of
same-meaning pairs that hit. By default the recommendation maximises F0.5, which
counts a wrong answer as worse than a miss. Pass target_precision=0.99 to get the
loosest threshold that is still right 99% of the time. result.mistakes() lists the
pairs it gets wrong. Use the same embedder you'll use in the cache, and aim for at
least a few dozen pairs drawn from real traffic.
Tracking savings
s = cache.stats
s.hit_rate # 0.42
s.cost_saved # 18.73 (USD)
s.cost_spent # 25.90
s.savings_rate # 0.42 share of the would-be bill avoided
s.tokens_saved # 3_912_000
s.by_model["claude-opus-5-5"].hit_rate
s.to_dict() # for your metrics pipeline
Money saved is the price of the original call, credited every time its answer is
reused. Prices come from a PriceBook:
from semcache import PriceBook
cache = SemanticCache(pricing={"gpt-4o-mini": (0.15, 0.60)}) # merged over defaults
cache = SemanticCache(pricing=PriceBook(default=(1.0, 3.0))) # fallback for unknown models
cache.set(prompt, answer, cost=0.0123) # or give the exact cost
Claude list prices are bundled (run semcache prices). Prices change, so check your
provider's pricing page. Models without a price still count toward hit rate but add
$0 to savings.
Persistence, expiry and size limits
cache = SemanticCache(
path="cache.db", # SQLite file; entries and stats survive restarts
ttl=7 * 24 * 3600, # entries expire after a week
max_entries=50_000, # least recently used entries are evicted
namespace="support-bot",
)
- Responses stored in SQLite must be JSON-serialisable. The client wrappers handle this for you.
- A cache file remembers which embedder built it and refuses to open with a different one, because their vectors aren't comparable.
- Use
with SemanticCache(...) as cache:or callcache.close()so the latest stats are written.
Sharing one cache between servers (Redis)
SQLite keeps the cache on one machine. To let several processes or servers reuse each other's answers, keep it in Redis (or any Redis-compatible server such as Valkey; no modules needed):
from semcache import SemanticCache
from semcache.redis_store import RedisStore
cache = SemanticCache(store=RedisStore("redis://localhost:6379/0"))
- Every process sees the others' answers immediately.
- Stats are shared too: each process adds its own counts to the totals in Redis,
so
cache.statsshows the hit rate and money saved across all of them. prefix="team-a:"keeps several caches apart in one database.- Search runs in each process: it downloads each vector once, then only fetches the list of entry ids per lookup. That is fast up to tens of thousands of entries per namespace.
CLI
semcache stats --db cache.db # hit rate, tokens and money saved
semcache stats --db cache.db --json
semcache list --db cache.db # most-hit entries
semcache prune --db cache.db --older-than 86400
semcache clear --db cache.db [--namespace support-bot] [--stats]
semcache prices # bundled price table
How it works
prompt ──► normalise ──► exact match in scope? ──yes──► HIT (similarity 1.0)
│ no
▼
embed ──► nearest neighbour in scope ──► ≥ threshold? ──yes──► HIT
│ no
▼
MISS ──► call LLM ──► set()
A scope is namespace + model; the client wrappers add a hash of the rest of the
request. Search is an exact cosine scan over a NumPy matrix per scope, fast enough for
tens of thousands of entries per scope.
When not to use it
- Answers that depend on time or live data ("what's the weather now"). Use a
ttl, or don't cache those calls. - Prompts where small wording changes matter, such as code with specific values or math with different numbers. Use a high threshold or a separate namespace.
- Personalised answers. Put a user or tenant id in
namespaceso answers never leak between users.
Development
pip install -e ".[dev]"
pytest -q
ruff check . && ruff format --check .
See CONTRIBUTING.md and CHANGELOG.md.
Roadmap
- Pluggable vector indexes (FAISS, hnswlib) for millions of entries
- Redis and Postgres/pgvector stores
- Streaming support in the client wrappers
- Optional LLM-judged verification of borderline hits
License
Metadata
Release files for semcacher 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| semcacher-0.1.0.tar.gz | 42.1 kB | With provenance | Details for semcacher-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| semcacher-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for semcacher-0.1.0-py3-none-any.whl |
Total release size: 78.5 kB
Release files / semcacher-0.1.0.tar.gz
| Download URL | semcacher-0.1.0.tar.gz |
|---|---|
| Size | 42.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
997ad3071b091ef2c8ca5c23de1c39a75dd3d327ce13075b38a2c20a75fc47c5
|
|
BLAKE2b-256 checksum How to use checksums |
3bc84bc76de686fba28e847b31ee4ae477a314abed228440b5ad3d1e5a7e2204
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / semcacher-0.1.0-py3-none-any.whl
| Download URL | semcacher-0.1.0-py3-none-any.whl |
|---|---|
| Size | 36.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
364a3ef708f6028f11435cfe47023186590f58fd6a88fb60f06c1515a7f9191f
|
|
BLAKE2b-256 checksum How to use checksums |
55cfc9474ea46dccdbfb1e24033559aef1962d03878e7cb3319829f956df3a82
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logSecurity for semcacher 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for semcacher-0.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.