Conduit

A self-hosted, OpenAI-compatible LLM gateway that makes LLM calls cheaper, more reliable and faster. Point any OpenAI-SDK app at it by changing one URL; Conduit adds semantic caching, complexity-based model routing, retries with circuit breakers and fallback, hedged requests, rate limiting, a spending cap, and per-request cost/latency analytics.

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="anything")
client.chat.completions.create(model="auto", messages=[{"role": "user", "content": "Hi!"}])

Results

Benchmark: 1,000 requests per configuration, Poisson arrivals at 40 req/s, three simulated providers (log-normal latency, 2% 5xx, 1% 429, 4% multi-second stalls), with the premium provider down for 30–50% of every run. Reproduce with python -m conduit bench && python -m conduit report.

• 92% lower cost than the baseline, 100% success through the outage (baseline: 76.1%).
• Hedging halves p99 (4.51 s → 2.14 s) for 22% more cost; only 3.7% of requests are hedged.
• Semantic cache: 2 wrong answers in 1,000 requests. A labelled threshold sweep picked 0.85: ~60% hit rate vs 34% for exact matching, at 0.33 wrong answers per 1,000.

Numbers come from simulated providers and a synthetic workload (80% repeated questions in varied phrasings, 20% one-off prompts); savings depend on how much of real traffic is simple or repeated.

How a request flows

client ─► HTTP server ─► 1 cache ──hit──► answer (0.15 ms exact / 2.6 ms semantic, $0)
                             │ miss
                             ▼
                         2 admit   token-bucket rate limit, spending cap
                         3 route   complexity tier → rank by price, EWMA latency, error rate
                         4 call    fallback chain · circuit breakers · @retry · hedging ─► providers
                         5 record  cache put · cost ledger · JSONL log ─► Pandas / Matplotlib reports

Features

• Two-level cache: SHA-256 exact match, then a NumPy semantic match (hashing-trick embeddings, query-time IDF weighting, cosine similarity, entity guard for numbers/names), scoped by system prompt, model and temperature; LRU + TTL; only low-temperature requests are cached.
• Routing: transparent complexity score → tier; capable providers ranked by weighted price, EWMA latency and error rate; weaker tiers kept as a last resort.
• Resilience: retries with full-jitter exponential backoff (only for retryable errors, honours Retry-After), per-provider circuit breakers, lazy fallback chain, streaming fallback before the first token.
• Hedged requests: race an equally capable backup when the primary is slow; the loser's cost is recorded as hedge waste.
• Admission control: thread-safe token bucket and a spending cap (cache hits still served).
• Providers: simulated (offline, injectable outages), any OpenAI-compatible API (OpenAI, Groq, Gemini, OpenRouter, ...), local servers (Ollama, LM Studio).
• Observability: JSONL request logs, /metrics, Pandas summaries, six Matplotlib charts.
• Zero heavy dependencies: standard-library HTTP; NumPy, Pandas, Matplotlib.

Quick start

pip install .
python -m pytest -q          # 83 tests
conduit stats                # cache + routing demo
conduit serve                # gateway on http://127.0.0.1:8080/v1
conduit tour                 # every course concept, demonstrated live

Installation from PyPI

pip install lochangowda10-conduit

Development install

git clone https://github.com/lochangowda10/Conduit.git
cd Conduit
pip install .

Real providers: cp config.example.json config.json, export GROQ_API_KEY / GEMINI_API_KEY, check model IDs and prices in each provider's docs, then python -m conduit serve --config config.json.

Design decisions

Project layout

conduit/            gateway package
  gateway.py        the pipeline, hedging, streaming
  cache.py          exact + semantic cache
  embeddings.py     hashing-trick embedder (+ optional sentence-transformers)
  router.py         strategies, complexity score, FallbackChain iterator
  resilience.py     @retry, @log_execution, CircuitBreaker
  ratelimit.py      TokenBucket
  ledger.py         thread-safe cost ledger
  telemetry.py      JSONL logs, CSV export
  analytics.py      Pandas summaries
  server.py         OpenAI-compatible HTTP server (SSE streaming)
  providers/        ABC + mixins, simulated, OpenAI-compatible, local
benchmarks/         workload generator, benchmark, cache sweep, charts
demos/              course concept tour, race-condition + GIL demo
tests/              83 pytest tests (incl. end-to-end over real HTTP)
results/            benchmark CSVs and charts

Limitations

Single process (state in memory); heuristic router calibrated on synthetic prompts; estimated token counts when a provider doesn't report usage; no request coalescing for concurrent identical misses; hashing embeddings can confuse long prompts that differ by one word.