Skip to main content

semcacher

A semantic cache for LLM calls.
If a new question means the same as an old one, return the saved answer.
Track your hit rate and how much money it saved.

CI Python License


Why

Users ask the same things in different words. An exact-match cache misses "How do I reverse a list in Python?" vs "python: how to reverse a list", so you pay for the same answer twice. semcacher embeds each prompt, finds the closest one already answered, and returns that answer when the two are similar enough. Every hit is counted in tokens and dollars, so you can see what the cache is worth.

  • Drop-in: wrap an OpenAI or Anthropic client, decorate a function, or call get / set yourself.
  • Shows savings: hit rate, exact vs semantic hits, tokens saved, money saved and spent, broken down per model.
  • Safe defaults: answers never cross models, namespaces, system prompts or conversation history. Truncated or refused responses are not cached.
  • Small: the core needs only numpy. Neural embedders are optional extras.
  • Persistent: one SQLite file holds entries and stats. TTL and LRU size limits are built in.

Install

pip install semcacher                       # core (numpy only)
pip install "semcacher[local]"              # + sentence-transformers, runs locally
pip install "semcacher[openai]"             # + OpenAI embeddings
pip install "semcacher[redis]"              # + shared cache in Redis
pip install "semcacher[hnsw]"               # + fast search for very large caches
pip install "semcacher[postgres]"           # + cache in PostgreSQL (pgvector)
pip install "semcacher[prometheus]"         # + Prometheus metrics
pip install "semcacher[otel]"               # + OpenTelemetry metrics

You install semcacher and import semcache:

from semcache import SemanticCache

From source:

pip install git+https://github.com/sjapanjots/Semantic_Cache_LLM

Quickstart

from semcache import SemanticCache, LLMResult

cache = SemanticCache(pricing={"demo-model": (2.50, 10.00)})  # USD per 1M input/output tokens

def call_llm(prompt):
    ...  # your real LLM call
    return LLMResult("answer", input_tokens=1200, output_tokens=400)

cache.get_or_call("How do I reverse a list in Python?", lambda: call_llm(...), model="demo-model")
cache.get_or_call("python: how to reverse a list",      lambda: call_llm(...), model="demo-model")  # cache hit

print(cache.stats)

Output of examples/quickstart.py, which needs no API key:

lookups                   6
hits                      3  (50.0%)
  exact                   1
  semantic                2
misses                    3
tokens saved          4,800
money saved         $0.0210
money spent         $0.0210
savings rate          50.0%

Usage

1. Wrap your client (no other code changes)

from anthropic import Anthropic
from semcache import SemanticCache, SentenceTransformerEmbedder
from semcache.integrations import wrap_anthropic

cache = SemanticCache(embedder=SentenceTransformerEmbedder(), path="cache.db")
client = wrap_anthropic(Anthropic(), cache)

client.messages.create(
    model="claude-opus-5-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "How should a Python dev learn Rust?"}],
)

wrap_openai(OpenAI(), cache) does the same for client.chat.completions.create and for the Responses API, client.responses.create(model=..., input=...): the input text (or the last user message) is matched by meaning, while instructions, earlier input items, images and files, previous_response_id and conversation must be identical. Only completed responses are stored; background, incomplete and failed ones are not. stream=True works too: a miss streams live and the finished response is saved; a hit is replayed as the SDK's own Responses stream events. Async clients (AsyncAnthropic, AsyncOpenAI) work too. On a hit you get back the SDK's normal response object, rebuilt from the cache.

How requests are matched:

Part of the request Treatment
Last user message Compared by meaning
Model Must match exactly
System prompt, earlier turns, tools, temperature, max tokens, ... Hashed; must match exactly
timeout, extra_headers, user, metadata Ignored
stream=True Cached too: a miss streams live and is saved at the end; a hit is replayed as a stream
Responses stopped by max_tokens / refusal / length / content_filter Not cached

Streaming. stream=True works with both wrappers, sync and async:

with client.messages.create(model="claude-opus-5-5", max_tokens=1024, stream=True,
                            messages=[{"role": "user", "content": question}]) as stream:
    for event in stream:          # live from the API on a miss, replayed on a hit
        ...

On a miss you get the provider's stream unchanged, and the answer is saved once the stream finishes normally; if you stop reading early, nothing is saved. On a hit the saved answer is replayed as the provider's own event types (each content block's text arrives in one event). Streamed and non-streamed requests share entries. For OpenAI, pass stream_options={"include_usage": True} so savings can be counted.

2. Decorate a function

@cache.cached(model="claude-sonnet-5-5")
def ask(prompt: str) -> LLMResult:
    response = my_llm(prompt)
    return LLMResult(response.text, input_tokens=response.in_tok, output_tokens=response.out_tok)

ask("How many days are in a leap year?")
ask("how many days does a leap year have")   # served from the cache

Works on async def functions too. Use key=lambda *a, **kw: ... to choose which argument is the prompt.

3. Call get / set yourself

answer = cache.get(prompt, model="claude-opus-5-5")
if answer is None:
    answer, usage = call_llm(prompt)
    cache.set(prompt, answer, model="claude-opus-5-5",
              input_tokens=usage.input, output_tokens=usage.output)

cache.lookup(...) returns a CacheHit with .response, .similarity, .exact, .cost_saved and the stored .entry.

Choosing an embedder and threshold

The threshold is the minimum cosine similarity for a semantic hit. Each embedder ships a conservative default; tune it on your own traffic.

Embedder Install Default threshold Notes
HashingEmbedder (default) core 0.70 Lexical (words and character n-grams). No model download. Catches rewording, not synonyms.
SentenceTransformerEmbedder() [local] 0.85 Real semantic matching, runs on CPU. all-MiniLM-L6-v2 by default.
OpenAIEmbedder() [openai] 0.90 text-embedding-3-small by default. Costs a small amount per lookup.
CallableEmbedder(fn, name) core 0.85 Bring any model: Voyage, Cohere, an in-house service...

Too many wrong answers? Raise the threshold. Too few hits? Lower it, or switch to a neural embedder. You can also override it per call: cache.lookup(p, threshold=0.95).

Calibrating the threshold from your own data

Instead of guessing, label some prompt pairs as same or different meaning and let calibrate pick the threshold:

from semcache import HashingEmbedder, SemanticCache, calibrate, load_pairs

pairs = load_pairs("examples/tuning_pairs.csv")   # CSV columns: a,b,same
result = calibrate(HashingEmbedder(), pairs)        # or target_precision=0.99
print(result.summary())
cache = SemanticCache(threshold=result.threshold)
embedder   hashing-v2-1024-3
pairs      20 (10 same, 10 different)
recommend  threshold=0.7087  (best F0.5)

threshold  precision   recall  wrong hits    F0.5
   0.6725      87.5%    70.0%           1   0.833
   0.6800     100.0%    70.0%           0   0.921
   0.7087     100.0%    70.0%           0   0.921  <-
   0.7099     100.0%    60.0%           0   0.882

Precision is the share of hits that are correct; recall is the share of same-meaning pairs that hit. By default the recommendation maximises F0.5, which counts a wrong answer as worse than a miss. Pass target_precision=0.99 to get the loosest threshold that is still right 99% of the time. result.mistakes() lists the pairs it gets wrong. Use the same embedder you'll use in the cache, and aim for at least a few dozen pairs drawn from real traffic.

Double-checking borderline matches with a cheap LLM

Some different questions look almost identical to an embedder: "convert 10 miles to km" vs "convert 20 miles to km". A verifier asks a small model whether the saved answer really answers the new question before it is reused:

from anthropic import Anthropic
from semcache import SemanticCache
from semcache.verifier import LLMVerifier, anthropic_judge

verifier = LLMVerifier(anthropic_judge(Anthropic()), verify_below=0.95)
cache = SemanticCache(verifier=verifier, threshold=0.6)
print(verifier.stats)   # checks, approved, rejected, judge tokens and cost
  • Only semantic hits are checked; exact matches never are. verify_below skips the check for very close matches.
  • Because the verifier filters out bad matches, you can lower the threshold and get more hits.
  • anthropic_judge uses Claude Haiku 5.5 at low effort by default; openai_judge(client, model) and any judge(system, user) -> str function work too.
  • Decisions are remembered per question and entry, so repeats cost nothing.
  • If the judge fails or answers unclearly, the hit is rejected (on_error="reject"): an extra LLM call is cheaper than a wrong answer.

Tracking savings

s = cache.stats
s.hit_rate          # 0.42
s.cost_saved        # 18.73  (USD)
s.cost_spent        # 25.90
s.savings_rate      # 0.42   share of the would-be bill avoided
s.tokens_saved      # 3_912_000
s.by_model["claude-opus-5-5"].hit_rate
s.to_dict()         # for your metrics pipeline

Money saved is the price of the original call, credited every time its answer is reused. Prices come from a PriceBook:

from semcache import PriceBook

cache = SemanticCache(pricing={"gpt-4o-mini": (0.15, 0.60)})        # merged over defaults
cache = SemanticCache(pricing=PriceBook(default=(1.0, 3.0)))        # fallback for unknown models
cache.set(prompt, answer, cost=0.0123)                              # or give the exact cost

Claude list prices are bundled (run semcache prices). Prices change, so check your provider's pricing page. Models without a price still count toward hit rate but add $0 to savings.

Dashboards: Prometheus and OpenTelemetry

semcache.metrics publishes the same numbers as cache.stats (lookups, exact and semantic hits, misses, tokens saved, money saved and spent, hit ratio, entries, and per-model versions) so you can graph and alert on them.

# Prometheus, with prometheus_client
from prometheus_client import REGISTRY, start_http_server
from semcache.metrics import PrometheusCollector

REGISTRY.register(PrometheusCollector(cache))
start_http_server(9100)                      # scrape http://host:9100/metrics

# Prometheus, no extra packages: return this text from your own /metrics route
from semcache.metrics import prometheus_text
body = prometheus_text(cache)

# OpenTelemetry: reported through whatever MeterProvider your app configured
from semcache.metrics import register_opentelemetry
register_opentelemetry(cache)

Values are read live on every scrape or export; there is nothing to update by hand. Main metric names: semcache_hits_total{kind="exact|semantic"}, semcache_cost_saved_usd_total, semcache_hit_ratio, semcache_model_hits_total{model="..."} (OpenTelemetry uses semcache.hits and so on).

Persistence, expiry and size limits

cache = SemanticCache(
    path="cache.db",        # SQLite file; entries and stats survive restarts
    ttl=7 * 24 * 3600,      # entries expire after a week
    max_entries=50_000,     # least recently used entries are evicted
    namespace="support-bot",
)
  • Responses stored in SQLite must be JSON-serialisable. The client wrappers handle this for you.
  • A cache file remembers which embedder built it and refuses to open with a different one, because their vectors aren't comparable.
  • Use with SemanticCache(...) as cache: or call cache.close() so the latest stats are written.

Sharing one cache between servers (Redis)

SQLite keeps the cache on one machine. To let several processes or servers reuse each other's answers, keep it in Redis (or any Redis-compatible server such as Valkey; no modules needed):

from semcache import SemanticCache
from semcache.redis_store import RedisStore

cache = SemanticCache(store=RedisStore("redis://localhost:6379/0"))
  • Every process sees the others' answers immediately.
  • Stats are shared too: each process adds its own counts to the totals in Redis, so cache.stats shows the hit rate and money saved across all of them.
  • prefix="team-a:" keeps several caches apart in one database.
  • Search runs in each process: it downloads each vector once, then only fetches the list of entry ids per lookup. That is fast up to tens of thousands of entries per namespace.

Keeping the cache in PostgreSQL (pgvector)

If you already run Postgres, keep the cache there: every server shares the same answers and stats, and the similarity search runs in the database.

from semcache import SemanticCache
from semcache.postgres_store import PostgresStore

cache = SemanticCache(store=PostgresStore("postgresql://user:pass@db:5432/app"))
  • Needs the pgvector extension; the store runs CREATE EXTENSION IF NOT EXISTS vector (or create it once yourself).
  • Creates semcache_entries (plus _meta and _stats tables) on the first write; table="team_a_cache" keeps several caches apart in one database.
  • index="hnsw" adds a pgvector HNSW index for very large tables. Search stays exact by default, and a namespace the approximate search misses is searched exactly.
  • Stats from all servers add up, as with Redis.

Very large caches (HNSW index)

By default every lookup compares the question with every saved entry: exact, and fast up to tens of thousands of entries per namespace. For more, use an HNSW index:

cache = SemanticCache(path="cache.db", index="hnsw")        # or InMemoryStore(index="hnsw")

Measured on 100,000 entries (128-dim vectors, 4 CPU cores):

per lookup finds the closest entry
exact (default) 1.35 ms always
index="hnsw" 0.30 ms 99.7%

HNSW is approximate, but a missed neighbour can only cost a cache miss, never a wrong answer: whatever it returns must still pass the similarity threshold. Tune it with index=lambda dim: HNSWIndex(dim, M=32, ef_search=256), or plug in any index with add / remove / search (see semcache.index). A SQLite cache rebuilds its index when it opens, in bulk (100,000 entries in about 12 s). RedisStore always searches exactly.

CLI

semcache stats --db cache.db          # hit rate, tokens and money saved
semcache stats --db cache.db --json
semcache list  --db cache.db          # most-hit entries
semcache prune --db cache.db --older-than 86400
semcache clear --db cache.db [--namespace support-bot] [--stats]
semcache prices                       # bundled price table

How it works

prompt ──► normalise ──► exact match in scope? ──yes──► HIT (similarity 1.0)
                              │ no
                              ▼
                           embed ──► nearest neighbour in scope ──► ≥ threshold? ──yes──► HIT
                                                                        │ no
                                                                        ▼
                                                                      MISS ──► call LLM ──► set()

A scope is namespace + model; the client wrappers add a hash of the rest of the request. Search is an exact cosine scan over a NumPy matrix per scope, fast enough for tens of thousands of entries per scope.

When not to use it

  • Answers that depend on time or live data ("what's the weather now"). Use a ttl, or don't cache those calls.
  • Prompts where small wording changes matter, such as code with specific values or math with different numbers. Use a high threshold or a separate namespace.
  • Personalised answers. Put a user or tenant id in namespace so answers never leak between users.

Development

pip install -e ".[dev]"
pytest -q
ruff check . && ruff format --check .

See CONTRIBUTING.md and CHANGELOG.md.

Roadmap

Everything planned so far has shipped. Ideas and requests are welcome: open an issue at https://github.com/sjapanjots/Semantic_Cache_LLM/issues.

License

Apache License 2.0. See NOTICE: if you share a modified version, keep the notices and mark the files you changed. Versions up to and including 0.1.0 were released under the MIT License.

Metadata

Release files for semcacher 0.4.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for semcacher 0.4.0
File Size Uploaded Provenance Actions
semcacher-0.4.0.tar.gz 95.6 kB With provenance Details for semcacher-0.4.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for semcacher 0.4.0
File Interpreter ABI Platform Actions
semcacher-0.4.0-py3-none-any.whl Python 3 none any Details for semcacher-0.4.0-py3-none-any.whl

Total release size: 167.0 kB

Release files / semcacher-0.4.0.tar.gz

Download URL semcacher-0.4.0.tar.gz
Size 95.6 kB
Tags Source
SHA-256 checksum
How to use checksums
005024493a3373e23d33b1c2848240afcad31d7c84bc499c9796084b3c3ef959
BLAKE2b-256 checksum
How to use checksums
d1e0110d0675f59587c131685af0f2596f601e74c4552d6358e0909b106d076e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Release files / semcacher-0.4.0-py3-none-any.whl

Download URL semcacher-0.4.0-py3-none-any.whl
Size 71.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8b90639b325815baea13080686f6d1ab0fd74aecaf848d1ac3ab8d194f83f11a
BLAKE2b-256 checksum
How to use checksums
546479cefd46953d747aa38138a706d1bd5c13939d2817fb5c2df6ee4487cdd3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Security for semcacher 0.4.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for semcacher-0.4.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page