Prospect AI: Semantic Cache & Reverse Proxy for LLM Inference

High-performance reverse proxy for LLM APIs. Combines two-tier semantic caching (exact-match + embedding-based similarity) to cut token costs by 8-11x and latency to under 15ms.

The Problem

LLM inference is expensive. Applications pay full token costs even for semantically duplicate queries:

• "How do I reset my password?"
• "I forgot my password, how to reset?"

Both queries mean the same thing, but standard caches achieve <5% hit rates on natural language.

Result: Wasted tokens, high costs, slow responses.

The Solution

Prospect AI is an intelligent gateway between your app and LLM providers (OpenAI, Anthropic, etc.). It:

1. Intercepts requests — Acts as a drop-in reverse proxy
2. Deduplicates — Exact-match cache (L1) + semantic similarity (L2)
3. Caches — Stores responses with embeddings
4. Returns — Cached response in <15ms

Result: 40-55% cache hit rate, 8-11x fewer tokens, <15ms latency on hits.

Installation

# Via pip
pip install prospect-cache-ai

# Via Docker
docker pull ghcr.io/craftedwithintent/prospect-ai:1.0.0

Quick Start

1. Start the Proxy

prospect-ai start --port 8000 --similarity 0.92

2. Point Your App to the Proxy

Before:

from openai import OpenAI
client = OpenAI(api_key="sk-...", base_url="https://api.openai.com/v1")

After:

from openai import OpenAI
client = OpenAI(api_key="sk-...", base_url="http://localhost:8000/v1")

3. Monitor Performance

prospect-ai stats

Output:

Cache Statistics:
  Total Requests: 1,248
  Cache Hits: 742 (59.4%)
  Avg Latency (Hit): 8.2ms
  Avg Latency (Miss): 1,850ms
  Cost Saved: $18.50

Key Features

Feature	Details
Two-Tier Cache	L1: Exact match (<1ms) + L2: Semantic similarity (<15ms)
OpenAI-Compatible	Drop-in proxy for /v1/chat/completions
Streaming Support	Full stream: true support with zero overhead
Smart Fallback	Automatic failover on 429/5xx errors
Storage Backends	In-memory, SQLite-vec (zero external dependencies)
CLI & Metrics	Live dashboard + Prometheus export
Production Ready	50+ tests, 80%+ coverage, strict typing

Performance

Scenario	Latency	Improvement
Cache Hit (L1 exact)	0.8ms	99% faster
Cache Hit (L2 semantic)	12ms	99% faster
Cache Miss (upstream)	1,200–3,500ms	—
Hit Rate	40–55%	8–11x token savings

Architecture

Your App (OpenAI SDK)
    ↓
Prospect Proxy
    ├─→ L1: Exact match hash lookup (<1ms)
    ├─→ L2: Semantic similarity check (<15ms)
    ├─→ Cache Miss: Forward to OpenAI/Anthropic/etc
    ├─→ Store response + embedding
    └─→ Return to app

Functional Core + Imperative Shell:

• Pure logic: similarity scoring, request normalization, fallback routing
• I/O: FastAPI async server, Async httpx gateway, pluggable storage

Usage Examples

Basic Cache Monitoring

# Start proxy
prospect-ai start --port 8000 --similarity 0.92

# Check stats
prospect-ai stats

# List cached entries
prospect-ai cache list

# Clear cache
prospect-ai cache clear

Docker Compose

version: '3.9'
services:
  prospect:
    image: ghcr.io/craftedwithintent/prospect-ai:latest
    ports:
      - "8000:8000"
    environment:
      OPENAI_API_KEY: ${OPENAI_API_KEY}
  
  app:
    build: .
    ports:
      - "9000:9000"
    environment:
      OPENAI_BASE_URL: "http://prospect:8000/v1"
    depends_on:
      - prospect

Full Production Example

Complete, tested example in examples/llm-chat-with-caching/:

• ✅ FastAPI chat service with caching
• ✅ Real-time metrics endpoint
• ✅ Cost tracking and savings calculation
• ✅ 14 comprehensive tests
• ✅ Docker + Kubernetes deployment

Run locally:

cd examples/llm-chat-with-caching
pip install -r requirements.txt
export OPENAI_API_KEY="your-key"

# Terminal 1: Start proxy
prospect-ai start --port 8000

# Terminal 2: Start app
python app.py

# Terminal 3: Make requests
curl -X POST http://localhost:9000/chat \
  -H "Content-Type: application/json" \
  -d '{"content": "What is AI?"}'

Configuration

Start Options

prospect-ai start [OPTIONS]

Options:
  --port PORT              Listening port (default: 8000)
  --similarity THRESHOLD   L2 threshold 0.0–1.0 (default: 0.92)
  --workers WORKERS        Worker threads (default: 4)
  --backend BACKEND        Cache backend: memory, sqlite-vec (default: memory)
  --log-level LEVEL        Logging: debug, info, warning, error (default: info)

cache.yaml (Optional)

server:
  port: 8000
  workers: 4

cache:
  similarity_threshold: 0.92
  backend: "memory"
  max_entries: 10000
  ttl_seconds: 3600

upstream:
  provider: "openai"
  api_key: "${OPENAI_API_KEY}"
  fallback:
    - provider: "anthropic"
      api_key: "${ANTHROPIC_API_KEY}"

Testing

# Install dev dependencies
pip install -e ".[dev]"

# Run all tests
pytest tests/ -v --cov=src/prospect_ai

# Run specific test suite
pytest tests/test_semantic_cache.py -v

Coverage: 80%+ (enforced)

Deployment

Local Development

pip install -e ".[dev]"
prospect-ai start --port 8000

Docker

docker run -p 8000:8000 \
  -e OPENAI_API_KEY=sk-... \
  ghcr.io/craftedwithintent/prospect-ai:1.0.0

Kubernetes

See docs/DEPLOYMENT.md for full K8s manifest.

Documentation

• Full README — Comprehensive guide
• Deployment Guide — Production setup
• Architecture — Design decisions
• CONTRIBUTING.md — Development guide
• Examples — Production-ready code

Support

• 🐛 GitHub Issues — Bug reports, features
• 💬 GitHub Discussions — Questions, ideas
• 📖 Full Repository — Source, docs, examples

License

MIT License — See LICENSE for details.

---

Completely standalone: Works with any OpenAI-compatible API. Part of the CraftedWithIntent ecosystem for production AI systems.