LLMSwitch

Free-tier AI provider router for developers. Write AI code once; LLMSwitch handles provider fallback across Groq, OpenRouter, NVIDIA, and more.

Install

pip install ai-switch

The distribution is named ai-switch; the import stays llmswitch.

Configure

Set an API key for any provider you want LLMSwitch to use — it only tries providers with a key present:

export GROQ_API_KEY="..."
export OPENROUTER_API_KEY="..."
export NVIDIA_API_KEY="..."

Usage

from llmswitch import AIClient

client = AIClient()
response = client.chat("Write a Python web scraper")
print(response)

Providers are tried in order (Groq, OpenRouter, NVIDIA by default). If one is rate-limited, unreachable, or errors, LLMSwitch automatically falls back to the next.

Model names

model accepts either a friendly alias ("fast-model", "smart-model" — see llmswitch/registry.py) that's resolved per-provider, or a raw provider-specific model name passed straight through:

client.chat("hello", model="fast-model")
client.chat("hello", model="llama-3.1-8b-instant")  # Groq-specific name

Health tracking

LLMSwitch remembers which providers just failed, so a rate-limited provider isn't retried on every call — it's rested and moved to the back of the queue:

client.chat("hi")      # groq is rate-limited -> falls back to openrouter
client.chat("hi")      # groq skipped entirely; openrouter served directly

Cooldowns respect the provider's own Retry-After header when present, and otherwise back off exponentially (60s, 120s, 240s… capped at 15 min for rate limits; shorter for transient outages). A success clears the streak.

If every provider is resting, LLMSwitch still tries them — closest-to-recovery first — rather than failing without an attempt.

Inspect current state at any time:

client.status()
# {'groq': {'available': False, 'cooldown_remaining': 57.0, ...},
#  'openrouter': {'available': True, 'total_successes': 2, ...}}

Choosing providers explicitly

from llmswitch import AIClient
from llmswitch.providers import GroqProvider, NVIDIAProvider

client = AIClient(providers=[GroqProvider(), NVIDIAProvider()])

See examples/ for more.

Tests

pip install -e ".[dev]"
pytest

Status

Pre-release (v0.1, MVP). See roadmap.md and todo.md for current progress.