LLMSwitch

Free-tier AI provider router for developers. Write AI code once; LLMSwitch handles provider fallback across Groq, OpenRouter, NVIDIA, and more.

Install

pip install ai-switch

The distribution is named ai-switch; the import stays llmswitch.

Configure

Set an API key for any provider you want LLMSwitch to use — it only tries providers with a key present:

export GROQ_API_KEY="..."
export CEREBRAS_API_KEY="..."
export GEMINI_API_KEY="..."
export OPENROUTER_API_KEY="..."
export NVIDIA_API_KEY="..."
export MISTRAL_API_KEY="..."
export HF_TOKEN="..."

Ollama as a local last resort

Ollama needs no API key and never rate-limits, so it makes a good final fallback when every cloud free tier is spent. It is opt-in — set OLLAMA_HOST to enable it, so users without a local server don't pay for a failed attempt on every request:

export OLLAMA_HOST="http://localhost:11434/v1"

Usage

from llmswitch import AIClient

client = AIClient()
response = client.chat("Write a Python web scraper")
print(response)

Providers are tried in order (Groq, OpenRouter, NVIDIA by default). If one is rate-limited, unreachable, or errors, LLMSwitch automatically falls back to the next.

Model names

model accepts either a friendly alias ("fast-model", "smart-model" — see llmswitch/registry.py) that's resolved per-provider, or a raw provider-specific model name passed straight through:

client.chat("hello", model="fast-model")
client.chat("hello", model="llama-3.1-8b-instant")  # Groq-specific name

Health tracking

LLMSwitch remembers which providers just failed, so a rate-limited provider isn't retried on every call — it's rested and moved to the back of the queue:

client.chat("hi")      # groq is rate-limited -> falls back to openrouter
client.chat("hi")      # groq skipped entirely; openrouter served directly

Cooldowns respect the provider's own Retry-After header when present, and otherwise back off exponentially (60s, 120s, 240s… capped at 15 min for rate limits; shorter for transient outages). A success clears the streak.

If every provider is resting, LLMSwitch still tries them — closest-to-recovery first — rather than failing without an attempt.

Inspect current state at any time:

client.status()
# {'groq': {'available': False, 'cooldown_remaining': 57.0, ...},
#  'openrouter': {'available': True, 'total_successes': 2, ...}}

Choosing providers explicitly

from llmswitch import AIClient
from llmswitch.providers import GroqProvider, NVIDIAProvider

client = AIClient(providers=[GroqProvider(), NVIDIAProvider()])

See examples/ for more.

Tests

pip install -e ".[dev]"
pytest

Status

Pre-release (v0.1, MVP). See roadmap.md and todo.md for current progress.