CacheLLM

A drop-in semantic cache for OpenAI-compatible LLM APIs. Change one base URL, and questions your model has already answered come back in milliseconds instead of seconds.

On a 2,000-request replay against AWS Bedrock it served 77% of traffic from cache with zero false positives on genuinely new questions, cutting spend by 78% and p95 latency from 1,023 ms to 5.7 ms on a cache hit, or 9.7 ms when the question was reworded.

Quick start · How it works · Evaluation · API reference · Deployment

Real requests against Amazon Nova Micro on AWS Bedrock. A new question takes 1.6 seconds. The same question repeated takes 2.5 ms. A reworded version takes 2.9 ms at 0.99 similarity. A genuinely different question still misses, and anything with personal data in it is never stored.

llm semantic-cache openai-compatible fastapi redis vector-search aws-bedrock llmops prometheus grafana opentelemetry cost-optimization

Status. Working software with a real test suite and reproducible numbers. It runs locally or in Docker. There is no hosted demo URL: this is infrastructure you run in front of your own LLM calls, so the quick start has it serving traffic in about two minutes.

The problem

Every team running LLMs at scale pays twice for the same answer. Users ask the same questions in different words, support bots field the same twenty issues all day, and batch jobs re-classify near-identical rows. Each of those is a fresh API call: real money, and one to three seconds a user waits.

A normal cache does not help. Change one letter and the key misses, so exact-match caching catches almost nothing in natural language.

CacheLLM caches by meaning. It embeds each prompt, searches for the nearest answer it has already paid for, and serves it when the match is close enough. Your application changes one line:

client = OpenAI(base_url="http://localhost:8080/v1")   # was https://api.openai.com/v1

Everything else stays the same: same request shape, same response shape, same errors, same streaming. Responses carry X-Cache headers so you can see exactly what happened.

Headline results

2,000 requests at concurrency 8 against Amazon Nova Micro on AWS Bedrock, from a laptop in India. Full method, caveats and reproduction steps in docs/evaluation.md.

Hit rate climbed from 44% in the first hundred requests to 87% in the last hundred as the cache warmed. The whole run cost $0.0038 in real Bedrock charges, because 1,540 of the 2,000 requests never reached the model.

The same benchmark runs without any cloud credentials against the built-in fake provider, and lands within half a point: 77.2% hit rate, again with zero false positives. That version is what CI asserts on every push.

Key features

Tech stack

How it works

System architecture

flowchart TB
    App["Your application<br/>(OpenAI SDK, LangChain,<br/>Open WebUI, curl)"]

    subgraph Proxy["CacheLLM proxy (FastAPI)"]
        direction TB
        Auth["Auth + policy<br/>cacheable? category? TTL?"]
        L1["Tier 1: exact match<br/>normalised hash"]
        Embed["Embedder<br/>MiniLM ONNX, in process"]
        L2["Tier 2: semantic search<br/>HNSW, cosine, namespace filtered"]
        Flight["Single-flight<br/>stampede guard"]
        Router["Provider router"]
        Auth --> L1 --> Embed --> L2 --> Flight --> Router
    end

    subgraph Redis["Redis 8"]
        Keys["Exact keys<br/>hash to entry id"]
        Index["Vector index<br/>entry + embedding + TTL"]
        Stats["Counters and<br/>near-miss log"]
    end

    subgraph Providers["Upstream models"]
        Bedrock["AWS Bedrock<br/>Nova, Claude, Llama, Mistral"]
        OpenAICompat["OpenAI-compatible<br/>OpenAI, Groq, vLLM, Together"]
    end

    subgraph Obs["Observability"]
        Prom["Prometheus"]
        Graf["Grafana"]
        Otel["OpenTelemetry<br/>to Langfuse or Tempo"]
    end

    App -->|"POST /v1/chat/completions"| Auth
    Proxy -.-> Redis
    Router --> Bedrock
    Router --> OpenAICompat
    Proxy -->|"/metrics"| Prom --> Graf
    Proxy -.->|"spans"| Otel
    Proxy -->|"response + X-Cache headers"| App

In plain language. Your app talks to CacheLLM exactly as it would talk to OpenAI. The proxy first decides whether this request may be cached at all. If it may, it tries a cheap exact-match lookup. Failing that, it embeds the prompt locally and asks Redis for the nearest answer it has already paid for, but only among answers allowed to serve this system prompt and model. If nothing is close enough, it forwards to the real provider, streams the answer back, and stores it for next time. Every step increments a metric, so hit rate and money saved are visible live.

Request flow

sequenceDiagram
    autonumber
    participant C as Client
    participant P as CacheLLM
    participant E as Embedder
    participant R as Redis
    participant M as Provider

    C->>P: POST /v1/chat/completions
    P->>P: Cacheable? Temperature, tools, JSON mode,<br/>multi-turn, personal data
    alt Not cacheable
        P->>M: Forward untouched
        M-->>C: Response, X-Cache: BYPASS
    else Cacheable
        P->>R: Tier 1, exact hash lookup
        alt Exact hit
            R-->>P: Stored answer
            P-->>C: Response in about 1 ms, X-Cache: HIT (exact)
        else No exact match
            P->>E: Embed prompt (about 6 ms)
            E-->>P: 384-dim unit vector
            P->>R: KNN inside this namespace
            R-->>P: Nearest neighbours with scores
            alt Best score >= calibrated threshold
                P->>R: Increment hit count
                P-->>C: Response, X-Cache: HIT (semantic), X-Cache-Similarity
            else Below threshold
                P->>R: Record near miss if close
                P->>P: Single-flight: join an identical call in progress?
                P->>M: Call the model
                M-->>P: Answer, tokens, finish reason
                alt Finished cleanly
                    P->>R: Store entry with TTL for its category
                end
                P-->>C: Response, X-Cache: MISS
            end
        end
    end

Why two tiers

The exact tier costs one Redis GET and no embedding. Against Bedrock it produced 1,265 of 1,540 hits: 82% of all cache hits, at about 2.6 ms each, with no possibility of a semantic mistake. The semantic tier added another 275 hits, close to 14 points of hit rate, and it is the tier that carries risk. Building the cheap safe tier first is the difference between a demo and something you would deploy.

What is deliberately not cached

Prompts above 0.3 temperature, requests for multiple completions, tool calls, JSON mode, multi-turn conversations, non-text content, prompts over 8,000 characters, and anything that looks like personal data. Each rule maps to a specific way a naive cache goes wrong, each is a flag you can flip, and each shows up in the response as X-Cache-Bypass-Reason.

The evaluation pipeline

This is where a caching project usually hand-waves. The full write-up is in docs/evaluation.md; here is the shape of it.

Data. A hand-built labelled corpus in eval/corpus.py: 40 question groups covering 120 prompts, plus 35 hard negatives. Hard negatives are pairs one word apart with opposite meaning, like "undo the last git commit" against "undo the last git merge". That gives 193 labelled pairs, 123 duplicates and 70 non-duplicates. Any cache scores well on paraphrases alone, so hard negatives are the actual test.

Method. Embed both sides of every pair, sweep the cosine threshold from 0.70 to 1.00, and report recall, precision, F1 and how many hard negatives got served at each step. Then repeat across six embedding models and score each on the metric that matters: the most recall you can get while serving zero hard negatives.

Result 1: one threshold cannot do the job. With bge-small, the model most tutorials reach for, the first threshold that serves zero hard negatives is 0.96, and it catches 8.1% of genuine duplicates. Meanwhile "What is CORS?" and "Explain cross origin resource sharing" score 0.55, while "git commit" against "git merge" scores 0.95. The distributions overlap.

Result 2: thresholds do not transfer between models. The safe threshold ranges from 0.89 for MiniLM to 0.98 for Arctic-embed. Every model tested had negative separation, meaning the mean duplicate score sat below the worst hard negative. Bigger and slower did not fix it.

MiniLM gives four times the safe recall of bge-small, for a slightly larger download of 86 MB against 63 MB, so it is the default. The embed column is a ranking from short synthetic strings, not what a real question costs. The load test measures that directly. The whole table ships in code as CALIBRATED_THRESHOLDS, and the proxy warns at startup if you configure a model it has never measured.

Result 3, a negative one: a lexical guard does not rescue it. The obvious fix is to require matched prompts to share content words. Measured, hard negatives have higher token overlap (0.42 mean) than genuine paraphrases share vocabulary, because they differ by exactly one decisive word. The guard rejects good matches and keeps dangerous ones. It was measured and dropped rather than shipped.

Result 4: the benchmark was flattering itself. The first load test reported 79.1% and also showed 37 supposedly-new questions hitting the cache. They were not new: the long-tail generator emitted five phrasings per topic, so the "unseen" pool was full of paraphrases of itself. Fixed to one phrasing per topic, cross-matching fell to zero and the headline dropped to 77.2%. The lower number is the honest one.

Quick start

Python 3.11 or newer, and nothing else.

pip install cachellm-proxy
cachellm serve

That is the whole install. No Redis, no Docker, no config file. On startup it looks for a provider you already have: an API key in your environment, Ollama running locally, or AWS credentials. It logs which one it picked and how to override it.

To see what it found before starting anything:

cachellm providers

To try it with no account at all, the built-in test double stands in for a model:

CACHELLM_DEFAULT_PROVIDER=fake CACHELLM_FAKE_LATENCY_MS=600 cachellm serve

You only install what you actually route to. The base package is the proxy plus the local embedding model; AWS and Redis are extras, and nothing here installs Grafana or a Prometheus server, which are separate programs rather than Python packages.

Ask for a provider or backend whose extra is missing and the proxy names the exact command to fix it rather than raising an import error.

The distribution is cachellm-proxy because PyPI blocks cachellm as too close to an existing cachelm. The import name and the CLI are both still cachellm.

From source, if you plan to change anything:

git clone https://github.com/adarshcod30/CacheLLM.git
cd CacheLLM
uv sync && uv run cachellm serve

In another terminal, ask the same thing twice and watch the second one come back instantly:

curl -s -D- http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"fake/echo","temperature":0,"messages":[{"role":"user","content":"What is Redis used for?"}]}' | grep -i '^x-cache'

Run it again, then try a reworded version, and compare the headers:

curl -s -D- http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"fake/echo","temperature":0,"messages":[{"role":"user","content":"explain what redis does"}]}' | grep -i '^x-cache'

You should see X-Cache: MISS, then HIT on the exact repeat with X-Cache-Tier: exact, then HIT on the reworded one with X-Cache-Tier: semantic and a similarity score.

Point a real client at it:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="unused")
response = client.chat.completions.create(
    model="fake/echo",
    temperature=0,
    messages=[{"role": "user", "content": "What is Redis used for?"}],
)
print(response.choices[0].message.content)

Run the whole stack, including Prometheus and a pre-provisioned Grafana dashboard:

make up

Then open Grafana at http://localhost:3000 and the API docs at http://localhost:8080/docs.

Reproduce the numbers in this README:

make tune && make compare-models && make bench

Using a real provider

One environment variable per host. Everything except Bedrock speaks the OpenAI protocol, so a single adapter covers all of it and needs no extra.

CACHELLM_OPENAI_BASE_URL=https://api.groq.com/openai/v1 \
CACHELLM_OPENAI_API_KEY=gsk_your_key \
cachellm serve

If the key is already in its usual variable, such as GROQ_API_KEY or GEMINI_API_KEY, you can skip both lines: the proxy finds it at startup and says which host it picked.

Checked live, not just in tests. On 2026-09-11 the proxy was run against real Google Gemini (gemini-2.5-flash, gemini-3.5-flash) and Groq (openai/gpt-oss-20b, openai/gpt-oss-120b), driven by the official OpenAI SDK, alongside the AWS Bedrock benchmark. Each run covers a miss, an exact hit, a reworded hit, a similar question that must not hit, a stream and its replay, a hot-temperature bypass and an unknown model's error, and checks that the API key never reaches the log. Results are in docs/evaluation.md, and bench/live_check.py reruns them with your own key. The other hosts share the same adapter and its tests, but have not yet been run against the real service.

AWS Bedrock is the one exception, because it does not speak the OpenAI protocol. It needs the aws extra, and then uses your existing AWS credentials with no vendor API key at all:

pip install "cachellm-proxy[aws]"
CACHELLM_DEFAULT_PROVIDER=bedrock AWS_REGION=us-east-1 cachellm serve

curl -s http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"bedrock/us.amazon.nova-micro-v1:0","temperature":0,"messages":[{"role":"user","content":"What is Redis used for?"}]}'

If your credentials come from aws login rather than static keys or an SSO profile, that provider needs the CRT extra, which the aws extra already includes. The proxy detects that case and says so in the error rather than passing along boto's version of the message.

How a model name gets routed

Three rules, in order. You never configure a model list.

1. An explicit prefix this proxy owns: bedrock/…, openai/…, fake/….
2. A recognisable vendor convention. Bedrock ids are always vendor.model, so amazon.nova-lite-v1:0 and us.anthropic.claude-3-haiku-20240307-v1:0 are identified with no prefix. OpenAI's own families (gpt-, o1, o3, text-embedding-) are identified the same way, whatever the default is.
3. Otherwise the configured default, which is the OpenAI-compatible adapter. Names like llama3.2, mixtral-8x7b-32768 and qwen2.5-coder:7b are served by Groq, Ollama, Together and OpenRouter alike, so the endpoint you configured is the only sensible answer.

Model ids that legitimately contain a slash, which OpenRouter and Together both use, are forwarded whole. Only this proxy's own prefixes are stripped, and openai/ only when the upstream is OpenAI itself: Groq, OpenRouter and Together name OpenAI's models openai/gpt-oss-120b, so to them the prefix is part of the id. Note that anthropic.claude-… with a dot is a Bedrock id while anthropic/claude-… with a slash is an OpenRouter id, and the router tells them apart.

Ask it directly if you are unsure:

curl -s localhost:8080/admin/route/meta-llama/Llama-3.3-70B-Instruct-Turbo
curl -s localhost:8080/admin/providers

Where the cache lives

One setting, CACHELLM_BACKEND, with three values. Most people never touch it.

Memory is not the slow option. Below roughly 100,000 entries it is the faster one: scanning 20,000 cached prompts takes 0.85 ms, while a Redis round trip alone costs 2 to 3 ms. Redis earns its place when several processes need one shared cache. Run four copies of the proxy on memory and you have four separate caches, each a quarter as warm.

Memory can survive a restart too. Give it a file, and it saves there on a clean shutdown and loads it back on start:

CACHELLM_MEMORY_SNAPSHOT_PATH=$HOME/.cachellm/cache.npz cachellm serve

Setting up Redis. Install the extra, then run a Redis 8 server:

pip install "cachellm-proxy[redis]"
brew install redis && brew services start redis    # macOS
docker run -d -p 6379:6379 redis:8-alpine          # anywhere with Docker

The server has two requirements, and both are checked at startup:

• It needs the search module, which is what stores and searches vectors. Redis 8 from Homebrew or the official Docker image includes it. Many Linux distribution packages ship an older Redis without it, so prefer the Docker image there. A hosted Redis works if its plan includes search.
• It has to be database 0, because Redis search cannot index any other. Keep several apps apart with CACHELLM_INDEX_NAME instead.

If either is missing, auto uses memory and says exactly why, both in a startup warning and at the top of cachellm stats. With CACHELLM_BACKEND=redis the proxy never switches storage behind your back. It keeps answering without a cache and reports itself degraded until Redis is fixed.

API reference

Chat completions

POST /v1/chat/completions accepts the OpenAI chat completions body unchanged, streaming included. Unknown fields pass through rather than erroring, because OpenAI adds parameters regularly and a proxy that rejects them is not a drop-in.

Every response carries headers explaining the decision:

Clients can steer per request with X-Cache-Control:

Operator endpoints

Tune the threshold against your own traffic:

curl -s http://localhost:8080/admin/threshold-sweep -H 'Content-Type: application/json' -d '{"pairs":[{"a":"how do I reset my password","b":"password reset steps","duplicate":true},{"a":"how do I enable 2FA","b":"how do I disable 2FA","duplicate":false}]}'

Command line

cachellm serve                 # run the proxy, auto-detecting a provider
cachellm providers             # every host, and what this machine can reach
cachellm stats                 # hit rate, savings, latency, recent requests
cachellm watch                 # follow requests live, like tail -f
cachellm config                # effective configuration
cachellm invalidate --all
cachellm tune pairs.jsonl

cachellm stats is the dashboard, in your terminal:

  CacheLLM · memory backend · all-MiniLM-L6-v2

  HIT RATE    77.0%  ████████████████░░░░░░  1,540 of 2,000 requests
             1,265 exact · 275 semantic · 460 missed · 0 bypassed

  SAVED      $0.0135   spent $0.0038  78% lower
             116,029 tokens never generated

  LATENCY    cached       2.6 ms p50 ·      5.7 ms p95
             uncached   797.0 ms p50 ·   1022.8 ms p95   179x faster

  CACHE      451 entries · 9 coalesced · 17 near misses

  time      result tier        latency  score  saved       prompt
  02:30:55  HIT    exact         1.2ms  1.000  +$0.000010  What is Redis used for?
  02:30:54  BYPASS             451.5ms    ·          ·     What is my order 1234…  (pii:long_digits)
  02:30:53  HIT    semantic      7.4ms  0.952  +$0.000011  What is Redis typically used for?
  02:30:53  MISS               454.8ms    ·          ·     How do I configure nginx for TLS?

Prometheus and Grafana are still there if you want history and alerting, but nothing needs them.

Configuration

Every setting is an environment variable prefixed CACHELLM_, or a line in .env. Copy .env.example to start. The ones that matter most:

Deployment and infrastructure

Local. make up runs proxy, Redis, Prometheus and Grafana with the dashboard already provisioned. The image bakes the embedding model in, so a cold container does not spend its first request downloading it.

One box. The proxy is a single stateless container plus Redis. It is comfortable on a 1 GB instance: the embedding model uses about 100 MB of RAM and the cache size is your choice. Memory sizing is roughly 1 KB per entry for a 384-dimensional vector plus the stored answer, so 100,000 entries fits in well under a gigabyte. Redis is configured with allkeys-lru so it degrades by evicting rather than by failing.

Scaling out. Run several proxy replicas against one Redis. All state lives in Redis, so replicas are interchangeable. The stampede guard is per process, so N replicas can produce up to N duplicate calls for the same brand-new prompt, which is still far better than uncoordinated.

CI/CD. GitHub Actions runs lint, mypy, and the test suite against a real Redis 8 service container across Python 3.11, 3.12 and 3.13. A separate job boots the proxy, replays 300 requests and fails the build if hit rate collapses or if any genuinely-new question gets a cache hit. A fourth job builds the Docker image and boots it against Redis to check it comes up healthy.

Monitoring. Prometheus scrapes /metrics every five seconds.

The dashboard that ships with the repo, filled by 2,500 real requests through Bedrock. Latency is on a log axis because hits and misses are three orders of magnitude apart, which a linear axis would flatten into a single line at zero. The shipped Grafana dashboard has eleven panels: cumulative money saved, cost reduction, hit rate, cache size, latency by outcome on a log scale, request rate by outcome, similarity distributions for hits against misses, hits by category, near misses and stampedes, tokens spent against avoided, and embedding time. Set CACHELLM_TRACING_ENABLED with an OTLP endpoint to send traces to Langfuse, Tempo or Jaeger.

Rolling it out safely. Turn on shadow mode, leave it for a week, read /admin/near-misses and /admin/stats, run /admin/threshold-sweep on pairs drawn from your own logs, then switch shadow mode off.

Project structure

cachellm/
├── src/cachellm/
│   ├── settings.py            # typed config, calibrated thresholds per model
│   ├── models.py              # OpenAI-shaped request and response schemas
│   ├── pricing.py             # token prices, overridable, drives money saved
│   ├── errors.py              # OpenAI-shaped error envelope
│   ├── cli.py                 # serve, stats, config, invalidate, tune
│   ├── api/
│   │   ├── app.py             # app assembly, health, metrics, error handlers
│   │   ├── routes_chat.py     # the drop-in endpoint, streaming both ways
│   │   ├── routes_admin.py    # stats, invalidation, near misses, sweep
│   │   ├── deps.py            # app state, fail-open wiring
│   │   ├── auth.py            # constant-time key check
│   │   └── sse.py             # server-sent event framing
│   ├── cache/
│   │   ├── keys.py            # normalisation, namespace, exact hash
│   │   ├── policy.py          # cacheable? category? TTL? personal data?
│   │   ├── exact_store.py     # tier 1
│   │   ├── vector_store.py    # tier 2, HNSW index and invalidation
│   │   ├── service.py         # lookup, store, invalidate, stats
│   │   ├── coalesce.py        # single-flight stampede guard
│   │   ├── analytics.py       # durable counters and the near-miss log
│   │   └── entry.py           # the stored record
│   ├── embeddings/            # fastembed backend, hashing backend for tests
│   ├── providers/             # base, bedrock, openai-compatible, fake, router
│   └── observability/         # prometheus metrics, optional OpenTelemetry
├── tests/                     # 126 tests, unit and integration
├── bench/
│   ├── workload.py            # realistic request mix, Zipf popularity
│   ├── replay.py              # the load test
│   ├── tune_threshold.py      # offline threshold sweep
│   └── compare_models.py      # six embedding models scored on safe recall
├── eval/corpus.py             # labelled paraphrases and hard negatives
├── dashboards/cachellm.json   # provisioned Grafana dashboard
├── deploy/                    # Prometheus and Grafana provisioning
├── docs/evaluation.md         # full method, results and negative results
├── results/                   # measured output behind every number here
├── compose.yaml
└── Dockerfile

Testing

make test          # 327 tests
make test-cov      # with coverage
make lint          # ruff and mypy

The suite covers cache key derivation and namespace isolation, every cacheability rule, the stampede guard under concurrency, Bedrock request translation, lossless SSE framing, and end-to-end cache behaviour against a real Redis. The compatibility test drives the proxy with the official openai Python SDK, non-streaming and streaming, which is what caught a bug where replayed cache hits silently dropped a space at every chunk boundary.

Tests use a deterministic hashing embedder, so CI needs no model download and results are identical on every machine.

Roadmap

• Verification pass on semantic hits. The measured finding is that no embedding model separates paraphrases from one-word-flipped opposites. The honest fix is a cheap second opinion: ask a small model whether the candidate and the incoming prompt are the same question, and only then serve. It costs a fraction of a generation and would let the threshold drop a long way.
• Cross-process stampede protection using a Redis lock, so replicas coordinate.
• In-process library mode, wrapping an OpenAI client directly for people who do not want to run a service.
• Per-tenant namespaces with a scope header, so multi-tenant apps can cache safely.
• Embeddings pass-through so /v1/embeddings can be cached too.
• Quora Question Pairs at scale for a 150,000-pair evaluation alongside the hand-built corpus.

Releasing

Tagging a version publishes to PyPI through Trusted Publishing, so no API token exists anywhere. See docs/publishing.md.

git tag v0.1.0 && git push origin v0.1.0

Contributing

Issues and pull requests are welcome. Please run make lint && make test before opening one. If you change anything that touches matching quality, include a threshold sweep in the description: the numbers matter more than the argument.

License

MIT. See LICENSE.

Contact

Adarsh Dwivedi GitHub · adarshdwivedi256@gmail.com