LLMSwitch

Free-tier AI provider router for developers. Write AI code once; LLMSwitch handles provider fallback across Groq, OpenRouter, NVIDIA, and more.

Install

pip install ai-switch

The distribution is named ai-switch; the import stays llmswitch.

Configure

Set an API key for any provider you want LLMSwitch to use — it only tries providers with a key present:

export GROQ_API_KEY="..."
export CEREBRAS_API_KEY="..."
export GEMINI_API_KEY="..."
export OPENROUTER_API_KEY="..."
export NVIDIA_API_KEY="..."
export MISTRAL_API_KEY="..."
export HF_TOKEN="..."

Provider	Env var	Order
Groq	GROQ_API_KEY	1
Cerebras	CEREBRAS_API_KEY	2
Gemini	GEMINI_API_KEY	3
OpenRouter	OPENROUTER_API_KEY	4
NVIDIA	NVIDIA_API_KEY	5
Mistral	MISTRAL_API_KEY	6
Hugging Face	HF_TOKEN	7
Ollama (local)	OLLAMA_HOST	8 — last resort

Ollama as a local last resort

Ollama needs no API key and never rate-limits, so it makes a good final fallback when every cloud free tier is spent. It is opt-in — set OLLAMA_HOST to enable it, so users without a local server don't pay for a failed attempt on every request:

export OLLAMA_HOST="http://localhost:11434/v1"

Usage

from llmswitch import AIClient

client = AIClient()
response = client.chat("Write a Python web scraper")
print(response)

Providers are tried in order (Groq, OpenRouter, NVIDIA by default). If one is rate-limited, unreachable, or errors, LLMSwitch automatically falls back to the next.

Model names

model accepts either a friendly alias ("fast-model", "smart-model" — see llmswitch/registry.py) that's resolved per-provider, or a raw provider-specific model name passed straight through:

client.chat("hello", model="fast-model")
client.chat("hello", model="llama-3.1-8b-instant")  # Groq-specific name

Health tracking

LLMSwitch remembers which providers just failed, so a rate-limited provider isn't retried on every call — it's rested and moved to the back of the queue:

client.chat("hi")      # groq is rate-limited -> falls back to openrouter
client.chat("hi")      # groq skipped entirely; openrouter served directly

Cooldowns respect the provider's own Retry-After header when present, and otherwise back off exponentially (60s, 120s, 240s… capped at 15 min for rate limits; shorter for transient outages). A success clears the streak.

If every provider is resting, LLMSwitch still tries them — closest-to-recovery first — rather than failing without an attempt.

Inspect current state at any time:

client.status()
# {'groq': {'available': False, 'cooldown_remaining': 57.0, ...},
#  'openrouter': {'available': True, 'total_successes': 2, ...}}

Streaming

from llmswitch import AIClient, StreamInterrupted

with AIClient() as client:
    try:
        for token in client.stream("Write a haiku about failover"):
            print(token, end="", flush=True)
    except StreamInterrupted as exc:
        print(f"\n[broke on {exc.provider}, kept: {exc.partial!r}]")

Fallback is transparent only up to the first token. Once tokens have been handed to you they can't be un-yielded, so a later failure raises StreamInterrupted carrying the partial text rather than silently restarting the answer on another provider mid-sentence.

Async

import asyncio
from llmswitch import AsyncAIClient

async def main():
    async with AsyncAIClient() as client:
        print(await client.chat("hello"))
        async for token in client.stream("count to five"):
            print(token, end="")

asyncio.run(main())

AsyncAIClient shares the sync client's routing core, so fallback, cooldowns, and retry behave identically — only the transport differs.

Connections and cleanup

Clients pool connections per provider, so closing them matters. Use a context manager, or call close() / await aclose() yourself:

with AIClient() as client:      # closes on exit
    client.chat("hi")

Retries

A timeout usually means one unlucky request rather than a dead provider, so LLMSwitch retries the same provider once before failing over. A 429 is never retried — the provider explicitly said stop.

Choosing providers explicitly

from llmswitch import AIClient
from llmswitch.providers import GroqProvider, NVIDIAProvider

client = AIClient(providers=[GroqProvider(), NVIDIAProvider()])

See examples/ for more.

Tests

pip install -e ".[dev]"
pytest

Changelog

See CHANGELOG.md for what changed between releases.

Status

v0.4.0 on PyPI — eight providers, health tracking, streaming, async, 150 tests. Still 0.x: the API may shift before 1.0, and five of the eight adapters have verified endpoints but have not completed a request with a real key. See roadmap.md for the plan and todo.md for current progress.