OmniCache

OmniCache is a lightweight, local caching proxy for Anthropic (Claude), OpenAI (GPT), and Google (Gemini) APIs.

When developing with AI agents (like Claude Code, Cursor, Aider, or custom LLM scripts), repeated prompts, test runs, and static file queries frequently make duplicate upstream API calls. OmniCache sits between your client and upstream providers to intercept matching requests locally in <1ms, saving API costs and eliminating remote network latency.

Installation

pip install omnicache-proxy

Quickstart

1. Start the Proxy Server

omnicache

By default, the proxy runs on http://localhost:8000. You can change the port with --port:

omnicache --port 8080

2. Connect Your Client

Claude Code (Terminal CLI)

Set the Anthropic base URL environment variable before running claude:

export ANTHROPIC_BASE_URL="http://localhost:8000"
claude

Python (OpenAI SDK)

Route the base_url parameter to the local proxy:

from openai import OpenAI

client = OpenAI(
    api_key="your-api-key",
    base_url="http://localhost:8000/v1"
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "How do I reverse a linked list in Python?"}]
)
print(response.choices[0].message.content)

Cursor / VS Code / Other Tools

In your tool's model settings, set the API Base URL to http://localhost:8000/v1.

Key Features

• Two-Tier Cache Engine:
  • L1 Exact Match (Trie Hash): Sub-0.05ms lookup for identical payloads.
  • L2 Semantic Match (Cosine Similarity): Matches semantically equivalent prompts using an in-memory 512-dimension vector projection.
• Agent Stream Replayer: Emulates natural token-streaming for cached responses so interactive CLIs (like Claude Code) stream smoothly without terminal glitches.
• Request Coalescing (SingleFlight): Deduplicates concurrent in-flight requests for the same prompt, making only one upstream call.
• Zero Configuration Persistence: Automatically writes cache snapshots to ~/.omnicache/omnicache.db (SQLite) so cache state survives restarts.
• Built-in CLI Utilities:
  • omnicache doctor: Checks database state, port bindings, and embedder health.
  • omnicache benchmark: Measures P50, P95, and P99 cache lookup latencies on your machine.
  • omnicache stats: Prints total tokens and cost savings directly to the console.
• Observability:
  • Web Dashboard: http://localhost:8000/dashboard
  • Prometheus Metrics: http://localhost:8000/metrics
  • CSV Ledger Export: http://localhost:8000/v1/cache/export

CLI Reference

# Start server with clean terminal output
omnicache

# Start on custom port and host
omnicache --host 0.0.0.0 --port 9000

# Run with verbose HTTP access logs
omnicache --verbose

# Run system diagnostic checks
omnicache doctor

# Run local micro-benchmarks (1,000 queries)
omnicache benchmark

# View current token and cost savings
omnicache stats

Configuration

OmniCache can be configured via command-line flags or environment variables (in your shell or a local .env file):

Running Tests

Run the test suite using pytest:

git clone https://github.com/13manmayarai-hash/omnicache-proxy.git
cd omnicache-proxy
pip install -e .
pytest tests/ -v

Documentation

• API Reference
• Architecture Overview
• Troubleshooting & FAQ
• Contributing Guidelines

License

MIT License. See LICENSE for details.