⚡ LLM Gateway & Circuit Breaker

A high-throughput, distributed LLM Gateway and Circuit Breaker designed to protect AI agents, async workers, and microservices from upstream LLM provider outages, rate limits ($429$), and latency spikes.

🌟 Key Features

• 🛡️ Atomic Redis Circuit Breaker (FSM): Lock-free finite state machine (CLOSED → OPEN → HALF_OPEN) powered by single-roundtrip Redis Lua scripts with probe throttling and clock-skew protection.
• ⏱️ Dual-Budget Distributed Rate Limiter: Atomic token bucket tracking both Requests Per Minute (RPM) and Tokens Per Minute (TPM) with $O(1)$ token estimation and all-or-nothing consumption.
• 🔀 Resilient Routing & Fallback Chains: Multi-tier failover chains across primary and secondary providers with safe pre-yield streaming failovers.
• ⚡ Tiered Caching Layer:
  • Tier 1: Sub-millisecond deterministic SHA-256 exact match cache in Redis.
  • Tier 2: Local semantic similarity cache using fastembed ONNX embeddings (no external API calls to check cache).
• 🔌 Unified Provider Adapters: Native connection pooling and normalized OpenAI completion shapes for OpenAI, Anthropic, Azure OpenAI, Ollama, and Mock/Offline.
• 📊 Prometheus Metrics & Observability: Real-time histograms and counters for request latencies, status codes, and provider health.
• 📦 Dual Distribution: Use as an embedded Python Library or run as a standalone FastAPI Docker Proxy.

🏗️ System Architecture

                                  ┌───────────────────────────────┐
                                  │   Client Request (OpenAI SDK) │
                                  └───────────────┬───────────────┘
                                                  │
                                          [ Auth & Metrics ]
                                                  │
                                                  ▼
                                     ┌─────────────────────────┐
                                     │     Gateway Router      │
                                     └────────────┬────────────┘
                                                  │
                 ┌────────────────────────────────┴────────────────────────────────┐
                 │                                                                 │
                 ▼                                                                 ▼
      ┌────────────────────┐                                             ┌───────────────────┐
      │   Exact Match      │──(Cache Hit)──► [ Return Cached Response ]  │  Semantic Cache   │
      │   SHA-256 Cache    │                                             │  (fastembed ONNX) │
      └────────────────────┘                                             └───────────────────┘
                 │ (Cache Miss)
                 ▼
      ┌────────────────────┐
      │  Circuit Breaker   │──(Open / Tripped)──► [ Fast 503 Rejection / Cascade Fallback ]
      │  Atomic Lua Script │
      └──────────┬─────────┘
                 │ (Allowed)
                 ▼
      ┌────────────────────┐
      │ Rate Limiter Bucket│──(Exceeded)────────► [ 429 / Cascade Fallback ]
      │  (RPM + TPM Budget)│
      └──────────┬─────────┘
                 │ (Granted)
                 ▼
     ┌─────────────────────────────────────────────────────────────┐
     │                     Provider Dispatch                       │
     │   [Primary: OpenAI]  ──► (Fail) ──►  [Fallback: Anthropic]   │
     └─────────────────────────────────────────────────────────────┘

📦 Installation

As a Python Library

# Core library
pip install llm-gateway

# With FastAPI standalone proxy dependencies
pip install "llm-gateway[proxy]"

# With local ONNX semantic cache dependencies
pip install "llm-gateway[semantic-cache]"

# With full development & test suite
pip install "llm-gateway[dev,proxy,semantic-cache]"

🚀 Quick Start

1. Using as a Python Library (Direct Asyncio)

import asyncio
from llm_gateway import GatewayConfig, GatewayRouter
from llm_gateway.core.providers.base import ChatCompletionRequest, ChatMessage

async def main():
    # 1. Define configuration with fallback routes
    config = GatewayConfig(
        providers={
            "openai-main": {
                "type": "openai",
                "api_key": "sk-...",
                "default_model": "gpt-4o",
                "circuit_breaker": {"failure_threshold": 3, "recovery_timeout_s": 30},
                "rate_limit": {"rpm": 500, "tpm": 100000},
            },
            "anthropic-fallback": {
                "type": "anthropic",
                "api_key": "sk-ant-...",
                "default_model": "claude-3-5-sonnet-20241022",
            },
        },
        routes={
            "default": {
                "primary": "openai-main",
                "fallbacks": ["anthropic-fallback"],
            }
        },
        redis={"url": "redis://localhost:6379/0", "fail_open": True},
    )

    # 2. Instantiate router
    router = GatewayRouter(config)

    # 3. Dispatch chat completion
    req = ChatCompletionRequest(
        model="gpt-4o",
        messages=[ChatMessage(role="user", content="Explain quantum computing in 10 words.")],
    )
    response = await router.route(req, route_name="default")
    print(response.choices[0].message.content)

    await router.close()

if __name__ == "__main__":
    asyncio.run(main())

2. Running as a Standalone Docker Proxy

Step 1: Create your gateway.yaml

providers:
  openai:
    type: openai
    api_key: "${OPENAI_API_KEY}"
    default_model: gpt-4o
    circuit_breaker:
      failure_threshold: 3
      recovery_timeout_s: 30
    rate_limit:
      rpm: 1000
      tpm: 200000

  anthropic:
    type: anthropic
    api_key: "${ANTHROPIC_API_KEY}"
    default_model: claude-3-5-sonnet-20241022

  mock-offline:
    type: mock
    default_model: mock-gpt-4o
    mock_response: "Hello from local mock LLM!"

routes:
  default:
    primary: openai
    fallbacks: [anthropic, mock-offline]

redis:
  url: "redis://localhost:6379/0"
  fail_open: true

auth:
  enabled: false

Step 2: Start the Stack with Docker Compose

docker compose up --build

• LLM Gateway Endpoint: http://localhost:8000
• Prometheus Metrics: http://localhost:9090

Step 3: Call with the OpenAI SDK

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="any-string",  # or your Bearer token if auth.enabled is true
)

response = client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": "Hello LLM Gateway!"}],
)

print(response.choices[0].message.content)

📡 API Endpoints

⚙️ Configuration Reference

🧪 Testing & Load Benchmarking

Automated Test Suite

# Run all 209 unit, integration, and load tests
pytest tests/unit/ tests/integration/ tests/load/ --cov=llm_gateway

Live Concurrency & Stress Testing

# Non-streaming stress test (1,000 requests, 50 concurrent workers)
python3 stress_test.py -n 1000 -c 50

# Streaming SSE stress test (200 requests, 20 concurrent workers)
python3 stress_test.py -n 200 -c 20 --stream

Export OpenAPI & JSON Schemas

python3 scripts/export_schema.py --all ./schemas

📄 License

This project is licensed under the MIT License.