CacheLLM

A drop-in semantic cache for OpenAI-compatible LLM APIs. Change one base URL, and questions your model has already answered come back in milliseconds instead of seconds.

On a 2,000-request replay against AWS Bedrock it served 77% of traffic from cache with zero false positives on genuinely new questions, cutting spend by 78% and p95 latency from 1,023 ms to 5.7 ms.

Quick start · How it works · Evaluation · API reference · Deployment

Real requests against Amazon Nova Micro on AWS Bedrock. A new question takes 1.6 seconds. The same question repeated takes 2.5 ms. A reworded version takes 2.9 ms at 0.99 similarity. A genuinely different question still misses, and anything with personal data in it is never stored.

llm semantic-cache openai-compatible fastapi redis vector-search aws-bedrock llmops prometheus grafana opentelemetry cost-optimization

Status. Working software with a real test suite and reproducible numbers. It runs locally or in Docker. There is no hosted demo URL: this is infrastructure you run in front of your own LLM calls, so the quick start has it serving traffic in about two minutes.

The problem

Every team running LLMs at scale pays twice for the same answer. Users ask the same questions in different words, support bots field the same twenty issues all day, and batch jobs re-classify near-identical rows. Each of those is a fresh API call: real money, and one to three seconds a user waits.

A normal cache does not help. Change one letter and the key misses, so exact-match caching catches almost nothing in natural language.

CacheLLM caches by meaning. It embeds each prompt, searches for the nearest answer it has already paid for, and serves it when the match is close enough. Your application changes one line:

client = OpenAI(base_url="http://localhost:8080/v1")   # was https://api.openai.com/v1

Everything else stays the same: same request shape, same response shape, same errors, same streaming. Responses carry X-Cache headers so you can see exactly what happened.

Headline results

2,000 requests at concurrency 8 against Amazon Nova Micro on AWS Bedrock, from a laptop in India. Full method, caveats and reproduction steps in docs/evaluation.md.

Hit rate climbed from 44% in the first hundred requests to 87% in the last hundred as the cache warmed. The whole run cost $0.0038 in real Bedrock charges, because 1,540 of the 2,000 requests never reached the model.

The same benchmark runs without any cloud credentials against the built-in fake provider, and lands within half a point: 77.2% hit rate, again with zero false positives. That version is what CI asserts on every push.

Key features

Tech stack

How it works

System architecture

flowchart TB
    App["Your application<br/>(OpenAI SDK, LangChain,<br/>Open WebUI, curl)"]

    subgraph Proxy["CacheLLM proxy (FastAPI)"]
        direction TB
        Auth["Auth + policy<br/>cacheable? category? TTL?"]
        L1["Tier 1: exact match<br/>normalised hash"]
        Embed["Embedder<br/>MiniLM ONNX, in process"]
        L2["Tier 2: semantic search<br/>HNSW, cosine, namespace filtered"]
        Flight["Single-flight<br/>stampede guard"]
        Router["Provider router"]
        Auth --> L1 --> Embed --> L2 --> Flight --> Router
    end

    subgraph Redis["Redis 8"]
        Keys["Exact keys<br/>hash to entry id"]
        Index["Vector index<br/>entry + embedding + TTL"]
        Stats["Counters and<br/>near-miss log"]
    end

    subgraph Providers["Upstream models"]
        Bedrock["AWS Bedrock<br/>Nova, Claude, Llama, Mistral"]
        OpenAICompat["OpenAI-compatible<br/>OpenAI, Groq, vLLM, Together"]
    end

    subgraph Obs["Observability"]
        Prom["Prometheus"]
        Graf["Grafana"]
        Otel["OpenTelemetry<br/>to Langfuse or Tempo"]
    end

    App -->|"POST /v1/chat/completions"| Auth
    Proxy -.-> Redis
    Router --> Bedrock
    Router --> OpenAICompat
    Proxy -->|"/metrics"| Prom --> Graf
    Proxy -.->|"spans"| Otel
    Proxy -->|"response + X-Cache headers"| App

In plain language. Your app talks to CacheLLM exactly as it would talk to OpenAI. The proxy first decides whether this request may be cached at all. If it may, it tries a cheap exact-match lookup. Failing that, it embeds the prompt locally and asks Redis for the nearest answer it has already paid for, but only among answers allowed to serve this system prompt and model. If nothing is close enough, it forwards to the real provider, streams the answer back, and stores it for next time. Every step increments a metric, so hit rate and money saved are visible live.

Request flow

sequenceDiagram
    autonumber
    participant C as Client
    participant P as CacheLLM
    participant E as Embedder
    participant R as Redis
    participant M as Provider

    C->>P: POST /v1/chat/completions
    P->>P: Cacheable? Temperature, tools, JSON mode,<br/>multi-turn, personal data
    alt Not cacheable
        P->>M: Forward untouched
        M-->>C: Response, X-Cache: BYPASS
    else Cacheable
        P->>R: Tier 1, exact hash lookup
        alt Exact hit
            R-->>P: Stored answer
            P-->>C: Response in about 1 ms, X-Cache: HIT (exact)
        else No exact match
            P->>E: Embed prompt (about 6 ms)
            E-->>P: 384-dim unit vector
            P->>R: KNN inside this namespace
            R-->>P: Nearest neighbours with scores
            alt Best score >= calibrated threshold
                P->>R: Increment hit count
                P-->>C: Response, X-Cache: HIT (semantic), X-Cache-Similarity
            else Below threshold
                P->>R: Record near miss if close
                P->>P: Single-flight: join an identical call in progress?
                P->>M: Call the model
                M-->>P: Answer, tokens, finish reason
                alt Finished cleanly
                    P->>R: Store entry with TTL for its category
                end
                P-->>C: Response, X-Cache: MISS
            end
        end
    end

Why two tiers

The exact tier costs one Redis GET and no embedding. Against Bedrock it produced 1,265 of 1,540 hits: 82% of all cache hits, at about 2.6 ms each, with no possibility of a semantic mistake. The semantic tier added another 275 hits, close to 14 points of hit rate, and it is the tier that carries risk. Building the cheap safe tier first is the difference between a demo and something you would deploy.

What is deliberately not cached

Prompts above 0.3 temperature, requests for multiple completions, tool calls, JSON mode, multi-turn conversations, non-text content, prompts over 8,000 characters, and anything that looks like personal data. Each rule maps to a specific way a naive cache goes wrong, each is a flag you can flip, and each shows up in the response as X-Cache-Bypass-Reason.

The evaluation pipeline

This is where a caching project usually hand-waves. The full write-up is in docs/evaluation.md; here is the shape of it.

Data. A hand-built labelled corpus in eval/corpus.py: 40 question groups covering 120 prompts, plus 35 hard negatives. Hard negatives are pairs one word apart with opposite meaning, like "undo the last git commit" against "undo the last git merge". That gives 193 labelled pairs, 123 duplicates and 70 non-duplicates. Any cache scores well on paraphrases alone, so hard negatives are the actual test.

Method. Embed both sides of every pair, sweep the cosine threshold from 0.70 to 1.00, and report recall, precision, F1 and how many hard negatives got served at each step. Then repeat across six embedding models and score each on the metric that matters: the most recall you can get while serving zero hard negatives.

Result 1: one threshold cannot do the job. With bge-small, the model most tutorials reach for, the first threshold that serves zero hard negatives is 0.96, and it catches 8.1% of genuine duplicates. Meanwhile "What is CORS?" and "Explain cross origin resource sharing" score 0.55, while "git commit" against "git merge" scores 0.95. The distributions overlap.

Result 2: thresholds do not transfer between models. The safe threshold ranges from 0.89 for MiniLM to 0.98 for Arctic-embed. Every model tested had negative separation, meaning the mean duplicate score sat below the worst hard negative. Bigger and slower did not fix it.

MiniLM gives four times the safe recall of bge-small at a third of the download size, so it is the default. The whole table ships in code as CALIBRATED_THRESHOLDS, and the proxy warns at startup if you configure a model it has never measured.

Result 3, a negative one: a lexical guard does not rescue it. The obvious fix is to require matched prompts to share content words. Measured, hard negatives have higher token overlap (0.42 mean) than genuine paraphrases share vocabulary, because they differ by exactly one decisive word. The guard rejects good matches and keeps dangerous ones. It was measured and dropped rather than shipped.

Result 4: the benchmark was flattering itself. The first load test reported 79.1% and also showed 37 supposedly-new questions hitting the cache. They were not new: the long-tail generator emitted five phrasings per topic, so the "unseen" pool was full of paraphrases of itself. Fixed to one phrasing per topic, cross-matching fell to zero and the headline dropped to 77.2%. The lower number is the honest one.

Quick start

You need Python 3.11 or newer and a Redis 8 instance. Redis 8 is required because the vector index needs the query engine, and it must be database 0 because Redis Search only indexes that one.

pip install cachellm-proxy

The distribution is cachellm-proxy because PyPI blocks cachellm as too close to an existing cachelm. The import name and the CLI are both still cachellm.

Or from source, which is what you want if you plan to change anything:

git clone https://github.com/adarshcod30/CacheLLM.git
cd CacheLLM
uv sync

Start Redis if you do not have one running:

docker run -d --name redis -p 6379:6379 redis:8-alpine

Run the proxy. With no configuration it uses the built-in fake provider, so you can see it working before wiring up a real model or spending anything:

CACHELLM_DEFAULT_PROVIDER=fake CACHELLM_FAKE_LATENCY_MS=600 uv run cachellm serve

In another terminal, ask the same thing twice and watch the second one come back instantly:

curl -s -D- http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"fake/echo","temperature":0,"messages":[{"role":"user","content":"What is Redis used for?"}]}' | grep -i '^x-cache'

Run it again, then try a reworded version, and compare the headers:

curl -s -D- http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"fake/echo","temperature":0,"messages":[{"role":"user","content":"explain what redis does"}]}' | grep -i '^x-cache'

You should see X-Cache: MISS, then HIT on the exact repeat with X-Cache-Tier: exact, then HIT on the reworded one with X-Cache-Tier: semantic and a similarity score.

Point a real client at it:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="unused")
response = client.chat.completions.create(
    model="fake/echo",
    temperature=0,
    messages=[{"role": "user", "content": "What is Redis used for?"}],
)
print(response.choices[0].message.content)

Run the whole stack, including Prometheus and a pre-provisioned Grafana dashboard:

make up

Then open Grafana at http://localhost:3000 and the API docs at http://localhost:8080/docs.

Reproduce the numbers in this README:

make tune && make compare-models && make bench

Using a real provider

AWS Bedrock needs no extra vendor keys and reaches Nova, Claude, Llama and Mistral through one API:

CACHELLM_DEFAULT_PROVIDER=bedrock AWS_REGION=us-east-1 uv run cachellm serve

If your credentials come from aws login rather than static keys or an SSO profile, install the CRT extra once, because that credential provider needs it:

uv sync --extra aws

The proxy detects that case and says so in the error rather than passing along boto's version of the message.

curl -s http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"bedrock/us.amazon.nova-micro-v1:0","temperature":0,"messages":[{"role":"user","content":"What is Redis used for?"}]}'

Any OpenAI-compatible endpoint works too, which covers OpenAI, Groq, Together, OpenRouter and a local vLLM:

CACHELLM_DEFAULT_PROVIDER=openai CACHELLM_OPENAI_BASE_URL=https://api.groq.com/openai/v1 CACHELLM_OPENAI_API_KEY=... uv run cachellm serve

API reference

Chat completions

POST /v1/chat/completions accepts the OpenAI chat completions body unchanged, streaming included. Unknown fields pass through rather than erroring, because OpenAI adds parameters regularly and a proxy that rejects them is not a drop-in.

Every response carries headers explaining the decision:

Clients can steer per request with X-Cache-Control:

Operator endpoints

Tune the threshold against your own traffic:

curl -s http://localhost:8080/admin/threshold-sweep -H 'Content-Type: application/json' -d '{"pairs":[{"a":"how do I reset my password","b":"password reset steps","duplicate":true},{"a":"how do I enable 2FA","b":"how do I disable 2FA","duplicate":false}]}'

Command line

uv run cachellm serve          # run the proxy
uv run cachellm stats          # cache statistics
uv run cachellm config         # effective configuration
uv run cachellm invalidate --all
uv run cachellm tune pairs.jsonl

Configuration

Every setting is an environment variable prefixed CACHELLM_, or a line in .env. Copy .env.example to start. The ones that matter most:

Deployment and infrastructure

Local. make up runs proxy, Redis, Prometheus and Grafana with the dashboard already provisioned. The image bakes the embedding model in, so a cold container does not spend its first request downloading it.

One box. The proxy is a single stateless container plus Redis. It is comfortable on a 1 GB instance: the embedding model uses about 100 MB of RAM and the cache size is your choice. Memory sizing is roughly 1 KB per entry for a 384-dimensional vector plus the stored answer, so 100,000 entries fits in well under a gigabyte. Redis is configured with allkeys-lru so it degrades by evicting rather than by failing.

Scaling out. Run several proxy replicas against one Redis. All state lives in Redis, so replicas are interchangeable. The stampede guard is per process, so N replicas can produce up to N duplicate calls for the same brand-new prompt, which is still far better than uncoordinated.

CI/CD. GitHub Actions runs lint, mypy, and the test suite against a real Redis 8 service container across Python 3.11, 3.12 and 3.13. A separate job boots the proxy, replays 300 requests and fails the build if hit rate collapses or if any genuinely-new question gets a cache hit. A fourth job builds the Docker image and boots it against Redis to check it comes up healthy.

Monitoring. Prometheus scrapes /metrics every five seconds.

The dashboard that ships with the repo, filled by 2,500 real requests through Bedrock. Latency is on a log axis because hits and misses are three orders of magnitude apart, which a linear axis would flatten into a single line at zero. The shipped Grafana dashboard has eleven panels: cumulative money saved, cost reduction, hit rate, cache size, latency by outcome on a log scale, request rate by outcome, similarity distributions for hits against misses, hits by category, near misses and stampedes, tokens spent against avoided, and embedding time. Set CACHELLM_TRACING_ENABLED with an OTLP endpoint to send traces to Langfuse, Tempo or Jaeger.

Rolling it out safely. Turn on shadow mode, leave it for a week, read /admin/near-misses and /admin/stats, run /admin/threshold-sweep on pairs drawn from your own logs, then switch shadow mode off.

Project structure

cachellm/
├── src/cachellm/
│   ├── settings.py            # typed config, calibrated thresholds per model
│   ├── models.py              # OpenAI-shaped request and response schemas
│   ├── pricing.py             # token prices, overridable, drives money saved
│   ├── errors.py              # OpenAI-shaped error envelope
│   ├── cli.py                 # serve, stats, config, invalidate, tune
│   ├── api/
│   │   ├── app.py             # app assembly, health, metrics, error handlers
│   │   ├── routes_chat.py     # the drop-in endpoint, streaming both ways
│   │   ├── routes_admin.py    # stats, invalidation, near misses, sweep
│   │   ├── deps.py            # app state, fail-open wiring
│   │   ├── auth.py            # constant-time key check
│   │   └── sse.py             # server-sent event framing
│   ├── cache/
│   │   ├── keys.py            # normalisation, namespace, exact hash
│   │   ├── policy.py          # cacheable? category? TTL? personal data?
│   │   ├── exact_store.py     # tier 1
│   │   ├── vector_store.py    # tier 2, HNSW index and invalidation
│   │   ├── service.py         # lookup, store, invalidate, stats
│   │   ├── coalesce.py        # single-flight stampede guard
│   │   ├── analytics.py       # durable counters and the near-miss log
│   │   └── entry.py           # the stored record
│   ├── embeddings/            # fastembed backend, hashing backend for tests
│   ├── providers/             # base, bedrock, openai-compatible, fake, router
│   └── observability/         # prometheus metrics, optional OpenTelemetry
├── tests/                     # 126 tests, unit and integration
├── bench/
│   ├── workload.py            # realistic request mix, Zipf popularity
│   ├── replay.py              # the load test
│   ├── tune_threshold.py      # offline threshold sweep
│   └── compare_models.py      # six embedding models scored on safe recall
├── eval/corpus.py             # labelled paraphrases and hard negatives
├── dashboards/cachellm.json   # provisioned Grafana dashboard
├── deploy/                    # Prometheus and Grafana provisioning
├── docs/evaluation.md         # full method, results and negative results
├── results/                   # measured output behind every number here
├── compose.yaml
└── Dockerfile

Testing

make test          # 126 tests
make test-cov      # with coverage
make lint          # ruff and mypy

The suite covers cache key derivation and namespace isolation, every cacheability rule, the stampede guard under concurrency, Bedrock request translation, lossless SSE framing, and end-to-end cache behaviour against a real Redis. The compatibility test drives the proxy with the official openai Python SDK, non-streaming and streaming, which is what caught a bug where replayed cache hits silently dropped a space at every chunk boundary.

Tests use a deterministic hashing embedder, so CI needs no model download and results are identical on every machine.

Roadmap

• Verification pass on semantic hits. The measured finding is that no embedding model separates paraphrases from one-word-flipped opposites. The honest fix is a cheap second opinion: ask a small model whether the candidate and the incoming prompt are the same question, and only then serve. It costs a fraction of a generation and would let the threshold drop a long way.
• Cross-process stampede protection using a Redis lock, so replicas coordinate.
• In-process library mode, wrapping an OpenAI client directly for people who do not want to run a service.
• Per-tenant namespaces with a scope header, so multi-tenant apps can cache safely.
• Embeddings pass-through so /v1/embeddings can be cached too.
• Quora Question Pairs at scale for a 150,000-pair evaluation alongside the hand-built corpus.

Releasing

Tagging a version publishes to PyPI through Trusted Publishing, so no API token exists anywhere. See docs/publishing.md.

git tag v0.1.0 && git push origin v0.1.0

Contributing

Issues and pull requests are welcome. Please run make lint && make test before opening one. If you change anything that touches matching quality, include a threshold sweep in the description: the numbers matter more than the argument.

License

MIT. See LICENSE.

Contact

Adarsh Dwivedi GitHub · adarshdwivedi256@gmail.com