Skip to main content

semcacher

A semantic cache for LLM calls.
If a new question means the same as an old one, return the saved answer.
Track your hit rate and how much money it saved.

CI Python License


Why

Users ask the same things in different words. An exact-match cache misses "How do I reverse a list in Python?" vs "python: how to reverse a list", so you pay for the same answer twice. semcacher embeds each prompt, finds the closest one already answered, and returns that answer when the two are similar enough. Every hit is counted in tokens and dollars, so you can see what the cache is worth.

  • Drop-in: wrap an OpenAI or Anthropic client, decorate a function, or call get / set yourself.
  • Shows savings: hit rate, exact vs semantic hits, tokens saved, money saved and spent, broken down per model.
  • Safe defaults: answers never cross models, namespaces, system prompts or conversation history. Truncated or refused responses are not cached.
  • Small: the core needs only numpy. Neural embedders are optional extras.
  • Persistent: one SQLite file holds entries and stats. TTL and LRU size limits are built in.

Install

pip install semcacher                       # core (numpy only)
pip install "semcacher[local]"              # + sentence-transformers, runs locally
pip install "semcacher[openai]"             # + OpenAI embeddings
pip install "semcacher[redis]"              # + shared cache in Redis
pip install "semcacher[hnsw]"               # + fast search for very large caches
pip install "semcacher[prometheus]"         # + Prometheus metrics
pip install "semcacher[otel]"               # + OpenTelemetry metrics

You install semcacher and import semcache:

from semcache import SemanticCache

From source:

pip install git+https://github.com/sjapanjots/Semantic_Cache_LLM

Quickstart

from semcache import SemanticCache, LLMResult

cache = SemanticCache(pricing={"demo-model": (2.50, 10.00)})  # USD per 1M input/output tokens

def call_llm(prompt):
    ...  # your real LLM call
    return LLMResult("answer", input_tokens=1200, output_tokens=400)

cache.get_or_call("How do I reverse a list in Python?", lambda: call_llm(...), model="demo-model")
cache.get_or_call("python: how to reverse a list",      lambda: call_llm(...), model="demo-model")  # cache hit

print(cache.stats)

Output of examples/quickstart.py, which needs no API key:

lookups                   6
hits                      3  (50.0%)
  exact                   1
  semantic                2
misses                    3
tokens saved          4,800
money saved         $0.0210
money spent         $0.0210
savings rate          50.0%

Usage

1. Wrap your client (no other code changes)

from anthropic import Anthropic
from semcache import SemanticCache, SentenceTransformerEmbedder
from semcache.integrations import wrap_anthropic

cache = SemanticCache(embedder=SentenceTransformerEmbedder(), path="cache.db")
client = wrap_anthropic(Anthropic(), cache)

client.messages.create(
    model="claude-opus-5-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "How should a Python dev learn Rust?"}],
)

wrap_openai(OpenAI(), cache) does the same for client.chat.completions.create. Async clients (AsyncAnthropic, AsyncOpenAI) work too. On a hit you get back the SDK's normal response object, rebuilt from the cache.

How requests are matched:

Part of the request Treatment
Last user message Compared by meaning
Model Must match exactly
System prompt, earlier turns, tools, temperature, max tokens, ... Hashed; must match exactly
timeout, extra_headers, user, metadata Ignored
stream=True Cached too: a miss streams live and is saved at the end; a hit is replayed as a stream
Responses stopped by max_tokens / refusal / length / content_filter Not cached

Streaming. stream=True works with both wrappers, sync and async:

with client.messages.create(model="claude-opus-5-5", max_tokens=1024, stream=True,
                            messages=[{"role": "user", "content": question}]) as stream:
    for event in stream:          # live from the API on a miss, replayed on a hit
        ...

On a miss you get the provider's stream unchanged, and the answer is saved once the stream finishes normally; if you stop reading early, nothing is saved. On a hit the saved answer is replayed as the provider's own event types (each content block's text arrives in one event). Streamed and non-streamed requests share entries. For OpenAI, pass stream_options={"include_usage": True} so savings can be counted.

2. Decorate a function

@cache.cached(model="claude-sonnet-5-5")
def ask(prompt: str) -> LLMResult:
    response = my_llm(prompt)
    return LLMResult(response.text, input_tokens=response.in_tok, output_tokens=response.out_tok)

ask("How many days are in a leap year?")
ask("how many days does a leap year have")   # served from the cache

Works on async def functions too. Use key=lambda *a, **kw: ... to choose which argument is the prompt.

3. Call get / set yourself

answer = cache.get(prompt, model="claude-opus-5-5")
if answer is None:
    answer, usage = call_llm(prompt)
    cache.set(prompt, answer, model="claude-opus-5-5",
              input_tokens=usage.input, output_tokens=usage.output)

cache.lookup(...) returns a CacheHit with .response, .similarity, .exact, .cost_saved and the stored .entry.

Choosing an embedder and threshold

The threshold is the minimum cosine similarity for a semantic hit. Each embedder ships a conservative default; tune it on your own traffic.

Embedder Install Default threshold Notes
HashingEmbedder (default) core 0.70 Lexical (words and character n-grams). No model download. Catches rewording, not synonyms.
SentenceTransformerEmbedder() [local] 0.85 Real semantic matching, runs on CPU. all-MiniLM-L6-v2 by default.
OpenAIEmbedder() [openai] 0.90 text-embedding-3-small by default. Costs a small amount per lookup.
CallableEmbedder(fn, name) core 0.85 Bring any model: Voyage, Cohere, an in-house service...

Too many wrong answers? Raise the threshold. Too few hits? Lower it, or switch to a neural embedder. You can also override it per call: cache.lookup(p, threshold=0.95).

Calibrating the threshold from your own data

Instead of guessing, label some prompt pairs as same or different meaning and let calibrate pick the threshold:

from semcache import HashingEmbedder, SemanticCache, calibrate, load_pairs

pairs = load_pairs("examples/tuning_pairs.csv")   # CSV columns: a,b,same
result = calibrate(HashingEmbedder(), pairs)        # or target_precision=0.99
print(result.summary())
cache = SemanticCache(threshold=result.threshold)
embedder   hashing-v2-1024-3
pairs      20 (10 same, 10 different)
recommend  threshold=0.7087  (best F0.5)

threshold  precision   recall  wrong hits    F0.5
   0.6725      87.5%    70.0%           1   0.833
   0.6800     100.0%    70.0%           0   0.921
   0.7087     100.0%    70.0%           0   0.921  <-
   0.7099     100.0%    60.0%           0   0.882

Precision is the share of hits that are correct; recall is the share of same-meaning pairs that hit. By default the recommendation maximises F0.5, which counts a wrong answer as worse than a miss. Pass target_precision=0.99 to get the loosest threshold that is still right 99% of the time. result.mistakes() lists the pairs it gets wrong. Use the same embedder you'll use in the cache, and aim for at least a few dozen pairs drawn from real traffic.

Double-checking borderline matches with a cheap LLM

Some different questions look almost identical to an embedder: "convert 10 miles to km" vs "convert 20 miles to km". A verifier asks a small model whether the saved answer really answers the new question before it is reused:

from anthropic import Anthropic
from semcache import SemanticCache
from semcache.verifier import LLMVerifier, anthropic_judge

verifier = LLMVerifier(anthropic_judge(Anthropic()), verify_below=0.95)
cache = SemanticCache(verifier=verifier, threshold=0.6)
print(verifier.stats)   # checks, approved, rejected, judge tokens and cost
  • Only semantic hits are checked; exact matches never are. verify_below skips the check for very close matches.
  • Because the verifier filters out bad matches, you can lower the threshold and get more hits.
  • anthropic_judge uses Claude Haiku 5.5 at low effort by default; openai_judge(client, model) and any judge(system, user) -> str function work too.
  • Decisions are remembered per question and entry, so repeats cost nothing.
  • If the judge fails or answers unclearly, the hit is rejected (on_error="reject"): an extra LLM call is cheaper than a wrong answer.

Tracking savings

s = cache.stats
s.hit_rate          # 0.42
s.cost_saved        # 18.73  (USD)
s.cost_spent        # 25.90
s.savings_rate      # 0.42   share of the would-be bill avoided
s.tokens_saved      # 3_912_000
s.by_model["claude-opus-5-5"].hit_rate
s.to_dict()         # for your metrics pipeline

Money saved is the price of the original call, credited every time its answer is reused. Prices come from a PriceBook:

from semcache import PriceBook

cache = SemanticCache(pricing={"gpt-4o-mini": (0.15, 0.60)})        # merged over defaults
cache = SemanticCache(pricing=PriceBook(default=(1.0, 3.0)))        # fallback for unknown models
cache.set(prompt, answer, cost=0.0123)                              # or give the exact cost

Claude list prices are bundled (run semcache prices). Prices change, so check your provider's pricing page. Models without a price still count toward hit rate but add $0 to savings.

Dashboards: Prometheus and OpenTelemetry

semcache.metrics publishes the same numbers as cache.stats (lookups, exact and semantic hits, misses, tokens saved, money saved and spent, hit ratio, entries, and per-model versions) so you can graph and alert on them.

# Prometheus, with prometheus_client
from prometheus_client import REGISTRY, start_http_server
from semcache.metrics import PrometheusCollector

REGISTRY.register(PrometheusCollector(cache))
start_http_server(9100)                      # scrape http://host:9100/metrics

# Prometheus, no extra packages: return this text from your own /metrics route
from semcache.metrics import prometheus_text
body = prometheus_text(cache)

# OpenTelemetry: reported through whatever MeterProvider your app configured
from semcache.metrics import register_opentelemetry
register_opentelemetry(cache)

Values are read live on every scrape or export; there is nothing to update by hand. Main metric names: semcache_hits_total{kind="exact|semantic"}, semcache_cost_saved_usd_total, semcache_hit_ratio, semcache_model_hits_total{model="..."} (OpenTelemetry uses semcache.hits and so on).

Persistence, expiry and size limits

cache = SemanticCache(
    path="cache.db",        # SQLite file; entries and stats survive restarts
    ttl=7 * 24 * 3600,      # entries expire after a week
    max_entries=50_000,     # least recently used entries are evicted
    namespace="support-bot",
)
  • Responses stored in SQLite must be JSON-serialisable. The client wrappers handle this for you.
  • A cache file remembers which embedder built it and refuses to open with a different one, because their vectors aren't comparable.
  • Use with SemanticCache(...) as cache: or call cache.close() so the latest stats are written.

Sharing one cache between servers (Redis)

SQLite keeps the cache on one machine. To let several processes or servers reuse each other's answers, keep it in Redis (or any Redis-compatible server such as Valkey; no modules needed):

from semcache import SemanticCache
from semcache.redis_store import RedisStore

cache = SemanticCache(store=RedisStore("redis://localhost:6379/0"))
  • Every process sees the others' answers immediately.
  • Stats are shared too: each process adds its own counts to the totals in Redis, so cache.stats shows the hit rate and money saved across all of them.
  • prefix="team-a:" keeps several caches apart in one database.
  • Search runs in each process: it downloads each vector once, then only fetches the list of entry ids per lookup. That is fast up to tens of thousands of entries per namespace.

Very large caches (HNSW index)

By default every lookup compares the question with every saved entry: exact, and fast up to tens of thousands of entries per namespace. For more, use an HNSW index:

cache = SemanticCache(path="cache.db", index="hnsw")        # or InMemoryStore(index="hnsw")

Measured on 100,000 entries (128-dim vectors, 4 CPU cores):

per lookup finds the closest entry
exact (default) 1.35 ms always
index="hnsw" 0.30 ms 99.7%

HNSW is approximate, but a missed neighbour can only cost a cache miss, never a wrong answer: whatever it returns must still pass the similarity threshold. Tune it with index=lambda dim: HNSWIndex(dim, M=32, ef_search=256), or plug in any index with add / remove / search (see semcache.index). A SQLite cache rebuilds its index when it opens, in bulk (100,000 entries in about 12 s). RedisStore always searches exactly.

CLI

semcache stats --db cache.db          # hit rate, tokens and money saved
semcache stats --db cache.db --json
semcache list  --db cache.db          # most-hit entries
semcache prune --db cache.db --older-than 86400
semcache clear --db cache.db [--namespace support-bot] [--stats]
semcache prices                       # bundled price table

How it works

prompt ──► normalise ──► exact match in scope? ──yes──► HIT (similarity 1.0)
                              │ no
                              ▼
                           embed ──► nearest neighbour in scope ──► ≥ threshold? ──yes──► HIT
                                                                        │ no
                                                                        ▼
                                                                      MISS ──► call LLM ──► set()

A scope is namespace + model; the client wrappers add a hash of the rest of the request. Search is an exact cosine scan over a NumPy matrix per scope, fast enough for tens of thousands of entries per scope.

When not to use it

  • Answers that depend on time or live data ("what's the weather now"). Use a ttl, or don't cache those calls.
  • Prompts where small wording changes matter, such as code with specific values or math with different numbers. Use a high threshold or a separate namespace.
  • Personalised answers. Put a user or tenant id in namespace so answers never leak between users.

Development

pip install -e ".[dev]"
pytest -q
ruff check . && ruff format --check .

See CONTRIBUTING.md and CHANGELOG.md.

Roadmap

  • Postgres/pgvector store
  • OpenAI Responses API in wrap_openai

License

Apache License 2.0. See NOTICE: if you share a modified version, keep the notices and mark the files you changed. Versions up to and including 0.1.0 were released under the MIT License.

Metadata

Release files for semcacher 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for semcacher 0.2.0
File Size Uploaded Provenance Actions
semcacher-0.2.0.tar.gz 83.1 kB With provenance Details for semcacher-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for semcacher 0.2.0
File Interpreter ABI Platform Actions
semcacher-0.2.0-py3-none-any.whl Python 3 none any Details for semcacher-0.2.0-py3-none-any.whl

Total release size: 145.7 kB

Release files / semcacher-0.2.0.tar.gz

Download URL semcacher-0.2.0.tar.gz
Size 83.1 kB
Tags Source
SHA-256 checksum
How to use checksums
6d0137fcc62a9a54c6761b95a772b4f9c874778ddd4c82ce327a192688963fc8
BLAKE2b-256 checksum
How to use checksums
97b91df46b1a96f97e0b9cd75533a86813e7d63995a19ab65d6c2ddb708bf7cd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Release files / semcacher-0.2.0-py3-none-any.whl

Download URL semcacher-0.2.0-py3-none-any.whl
Size 62.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b85cee7722022caee7892669725b85eb5736e047dbf24997f401bd9520a43097
BLAKE2b-256 checksum
How to use checksums
988ca218687fc9ed142997d86520fc5fd41564d44122ef2dbadfc5c05b483a11
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Security for semcacher 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for semcacher-0.2.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page