llm-cache-proxy
One env var. 60–80% cheaper dev loops. A localhost proxy that caches identical OpenAI and Anthropic API calls on disk and replays them for free. Works with every existing tool.
Why
You're iterating on a prompt. You run the same call 40 times tweaking wording. That's 40× the spend on identical requests.
Or: you have a long-running script that re-fetches the same tool definitions every run during development. Or: your test suite calls the API.
llm-cache-proxy sits on localhost, speaks the OpenAI and Anthropic REST
protocols, and caches every successful response in a single SQLite file.
Identical requests (same method, path, body) get served from disk —
no network call, no spend.
It works with every tool because you only change one env var:
export OPENAI_BASE_URL=http://127.0.0.1:9001/openai/v1
export ANTHROPIC_BASE_URL=http://127.0.0.1:9001/anthropic
Cursor, Claude Code, your scripts, your tests, the OpenAI Python SDK, the Anthropic SDK — they all start using the cache automatically.
Install & run
pip install llm-cache-proxy
llm-cache-proxy
# or
uvx llm-cache-proxy
Default port: 9001. Default cache: ~/.cache/llm-cache-proxy/cache.db.
Use it
In whatever shell launches your tool / script:
# OpenAI
export OPENAI_BASE_URL=http://127.0.0.1:9001/openai/v1
# Anthropic
export ANTHROPIC_BASE_URL=http://127.0.0.1:9001/anthropic
# now run anything — Cursor, your script, pytest, etc.
Responses include a X-LLM-Cache: HIT|MISS header so you can see what
happened.
Bypass the cache for a single request:
curl -H "x-llm-cache-bypass: 1" http://127.0.0.1:9001/openai/v1/chat/completions ...
See what you saved
curl http://127.0.0.1:9001/stats
{
"hits": 312,
"misses": 87,
"bytes_served_from_cache": 4_182_404,
"entries": 87,
"cached_response_bytes": 1_205_211,
"by_model": {"gpt-4o": 41, "claude-sonnet-4": 46}
}
Clear the cache:
curl -X DELETE http://127.0.0.1:9001/cache
curl -X DELETE http://127.0.0.1:9001/stats
Config (all optional)
| Env var | Default | Description |
|---|---|---|
LLM_CACHE_PORT |
9001 |
Listen port. |
LLM_CACHE_HOST |
127.0.0.1 |
Listen host. |
LLM_CACHE_DIR |
~/.cache/llm-cache-proxy |
Where to put the SQLite file. |
LLM_CACHE_TTL |
0 |
TTL in seconds (0 = forever). |
LLM_CACHE_TIMEOUT |
300 |
Upstream request timeout. |
OPENAI_UPSTREAM |
https://api.openai.com |
Override the upstream. |
ANTHROPIC_UPSTREAM |
https://api.anthropic.com |
Override the upstream. |
Per-request:
- Header
x-llm-cache-bypass: 1— skip both read and write for this call. - Header
x-llm-cache-extra-key: <string>— add an extra dimension to the cache key (e.g., a user id, a session id).
How the cache key is built
sha256(method + "|" + path + "|" + body + optional extra_key)
Method + path + body is enough to make identical requests collide
deterministically. Headers are not included in the default key (so API
key rotation doesn't invalidate the cache) — set x-llm-cache-extra-key
if you want extra dimensions.
Only 2xx responses are cached. Errors always go through.
Caveats
- Streaming responses: when the upstream returns
text/event-stream, the full SSE body is captured and replayed verbatim on cache hit. That works but you lose per-token streaming feel. - Tool / function-calling responses cache fine — the whole completion object is one entry.
- Don't expose this proxy to the public internet — it has no auth and your API key flows through it.
Companion projects
- mcp-rec — VCR for MCP servers (similar idea, MCP layer).
- ai-first-scraper — clean Markdown for LLM agents.
About the author
Built by yubinkim444, who also makes Kay's Records — an app for iOS and Android.
If this project saved you time, giving the app a try is the nicest way to say thanks.
License
MIT © yubinkim444
Metadata
Release files for llm-cache-proxy 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| llm_cache_proxy-0.1.1.tar.gz | 7.7 kB | Without provenance | Details for llm_cache_proxy-0.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| llm_cache_proxy-0.1.1-py3-none-any.whl | Python 3 | none | any | Details for llm_cache_proxy-0.1.1-py3-none-any.whl |
Total release size: 16.3 kB
Release files / llm_cache_proxy-0.1.1.tar.gz
| Download URL | llm_cache_proxy-0.1.1.tar.gz |
|---|---|
| Size | 7.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
803ce0b83512ac6c7b40763a6f549e01e4acf0a6ac5823b1debf050b94e0bf7c
|
|
BLAKE2b-256 checksum How to use checksums |
8571ce97a6f53e64d6e51a985f6d4af474fea4e730301422a905b0ca9e423009
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.6
|
Release files / llm_cache_proxy-0.1.1-py3-none-any.whl
| Download URL | llm_cache_proxy-0.1.1-py3-none-any.whl |
|---|---|
| Size | 8.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6e1774fa122aadb902348ba344121aa22e29f05f61ef12b3084f936b275ce765
|
|
BLAKE2b-256 checksum How to use checksums |
fa7cf76356580d512a4deb8cd1a0f97c47bf1a3a453dcd59296706656047ae5c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.6
|
Security for llm-cache-proxy 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.