OmniCache
OmniCache is a lightweight, local caching proxy for Anthropic (Claude), OpenAI (GPT), and Google (Gemini) APIs.
When developing with AI agents (like Claude Code, Cursor, Aider, or custom LLM scripts), repeated prompts, test runs, and static file queries frequently make duplicate upstream API calls. OmniCache sits between your client and upstream providers to intercept matching requests locally in <1ms, saving API costs and eliminating remote network latency.
Installation
pip install omnicache-proxy
Quickstart
1. Start the Proxy Server
omnicache
By default, the proxy runs on http://localhost:8000. You can change the port with --port:
omnicache --port 8080
2. Connect Your Client
Claude Code (Terminal CLI)
Set the Anthropic base URL environment variable before running claude:
export ANTHROPIC_BASE_URL="http://localhost:8000"
claude
Python (OpenAI SDK)
Route the base_url parameter to the local proxy:
from openai import OpenAI
client = OpenAI(
api_key="your-api-key",
base_url="http://localhost:8000/v1"
)
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "How do I reverse a linked list in Python?"}]
)
print(response.choices[0].message.content)
Cursor / VS Code / Other Tools
In your tool's model settings, set the API Base URL to http://localhost:8000/v1.
Key Features
- Two-Tier Cache Engine:
- L1 Exact Match (Trie Hash / Redis): Sub-0.05ms lookup for identical payloads.
- L2 Semantic Match (Cosine Similarity): Matches semantically equivalent prompts using an in-memory or Redis-clustered 512-dimension vector projection.
- Horizontal Scaling & Redis Clustering:
- Pluggable storage adapter architecture supporting both zero-dependency standalone mode and distributed multi-worker/multi-replica clusters.
- Atomic spend tracking (
INCRBYFLOAT) and sliding-window Redis RPM rate limiting across all worker processes. - Synchronized cluster-wide Circuit Breaker & upstream model failover state.
- Agent Stream Replayer: Emulates natural token-streaming for cached responses so interactive CLIs (like Claude Code) stream smoothly without terminal glitches.
- Request Coalescing (SingleFlight): Deduplicates concurrent in-flight requests for the same prompt, making only one upstream call.
- Zero Configuration Persistence: Automatically writes cache snapshots to
~/.omnicache/omnicache.db(SQLite WAL mode) or Redis backend. - Built-in CLI Utilities:
omnicache doctor: Checks database state, port bindings, and embedder health.omnicache benchmark: Measures P50, P95, and P99 cache lookup latencies on your machine.omnicache stats: Prints total tokens and cost savings directly to the console.
- Observability:
- Web Dashboard:
http://localhost:8000/dashboard - Prometheus Metrics:
http://localhost:8000/metrics - CSV Ledger Export:
http://localhost:8000/v1/cache/export
- Web Dashboard:
Horizontal Multi-Worker Deployment
To run OmniCache with multiple worker processes or in a clustered container environment, simply provide REDIS_URL:
# Multi-worker deployment with Redis distributed state
REDIS_URL="redis://127.0.0.1:6379/0" uvicorn server.gateway:app --host 127.0.0.1 --port 8000 --workers 4
Configuration
OmniCache can be configured via command-line flags or environment variables (in your shell or a local .env file):
| Environment Variable | Default | Description |
|---|---|---|
HOST |
127.0.0.1 |
Host interface to listen on (local-first by default). |
PORT |
8000 |
Port to bind the proxy server to. |
REDIS_URL |
"" |
Redis connection URL (e.g. redis://127.0.0.1:6379/0) for multi-worker state clustering. |
CACHE_STORAGE_BACKEND |
auto |
Storage engine backend: auto, redis, or memory. |
REQUIRE_AUTH |
false |
When true, enforces valid API key registration on all requests. |
ADMIN_API_KEY |
"" |
Master admin secret for managing /v1/enterprise/quotas and data exports. |
PRIVACY_SALT |
(auto-generated) |
256-bit cryptographic salt for anonymized PII tokenization. |
SEMANTIC_CACHE_TTL_SECONDS |
604800 |
Default time-to-live for cache entries (7 days). |
SEMANTIC_SIMILARITY_THRESHOLD |
0.92 |
Minimum cosine similarity required for an L2 semantic cache hit. |
OMNICACHE_DB_PATH |
~/.omnicache/omnicache.db |
Path to SQLite persistence database (WAL mode enabled). |
ANTHROPIC_API_KEY |
(Optional) | Default upstream Anthropic API key (if not passed in client headers). |
OPENAI_API_KEY |
(Optional) | Default upstream OpenAI API key (if not passed in client headers). |
GEMINI_API_KEY |
(Optional) | Default upstream Google Gemini API key. |
Running Tests
Run the test suite using pytest:
git clone https://github.com/13manmayarai-hash/omnicache-proxy.git
cd omnicache-proxy
pip install -e .
pytest tests/ -v
Documentation
License
MIT License. See LICENSE for details.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file omnicache_proxy-2.3.0.tar.gz.
File metadata
- Download URL: omnicache_proxy-2.3.0.tar.gz
- Upload date:
- Size: 70.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
python-httpx/0.28.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
98a46f2ec1ffd618a26609e34109296c91964945823796f5cd7df359c9e1afd0
|
|
| MD5 |
4c6f606c034f27240dcc6a95ce90574c
|
|
| BLAKE2b-256 |
ec3be59f1dd01b2af31973486c01b94dfb543a6c08079f02b2ab0956c610ce4b
|
File details
Details for the file omnicache_proxy-2.3.0-py3-none-any.whl.
File metadata
- Download URL: omnicache_proxy-2.3.0-py3-none-any.whl
- Upload date:
- Size: 64.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
python-httpx/0.28.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
121c007db18c39b1f01a66711154e1cab01f00c94236ee6164b566636fdfc9c3
|
|
| MD5 |
2d62eeb505fba04de98c3502a388ff6e
|
|
| BLAKE2b-256 |
1f0b37a8b8f27ddae978638be8779e79c69de638483994eafab941b92ff1e0bb
|
