llm-failover

Multi-provider LLM fallback with rate-limit handling and automatic retry. Zero vendor lock-in.

Born from running 4 AI agents daily across 5 LLM providers in production at StaaS Fund. When one provider goes down or hits a rate limit, the next one picks up seamlessly.

Features

• Automatic failover — tries providers in order, moves to the next on failure
• Rate-limit detection — catches 429 errors and retries with exponential backoff
• Daily quota tracking — when a provider hits its daily cap, it's skipped for the rest of the session
• Payment failures — a 402 (credits spent or lapsed) is never retried; the provider is skipped for the session
• Empty answers fail over — a provider that answers with nothing (reasoning models can) moves on to the next one instead of returning ""
• Answer validation — pass accept= to fail over on answers you can't use, not just on errors
• Round-robin rotation — spreads calls across providers evenly instead of always hitting the first one
• Per-provider budgets — voluntarily rotate before hitting rate limits (e.g., limit Groq to 8 calls per session)
• Named chains — define "fast", "reliable", "morning" chains with different provider orders
• Blackout callback — hook into total failure events (e.g., alert Discord, PagerDuty)
• Bring your own providers — any (prompt, max_tokens) -> str function works
• Built-in helpers — pre-built functions for Groq, Gemini, Anthropic (Claude), Mistral, NVIDIA NIM, GitHub Models, SambaNova, Cloudflare Workers AI, OpenRouter
• Zero required dependencies — core library is pure Python. Provider SDKs are optional.

Install

pip install llm-failover-chain

With built-in provider helpers:

pip install "llm-failover-chain[groq,gemini]"

Or straight from GitHub: pip install git+https://github.com/agilepeter/llm-failover.git (pin a release with @v0.6.1).

Not pip install llm-failover. That name on PyPI belongs to a different project, and so does an upgrade by that name. This package is llm-failover-chain and imports as llm_failover_chain.

Upgrading from 0.5.0 or earlier: the import name changed in 0.6.0. Replace llm_failover with llm_failover_chain in your imports (and in any logging config). Installed before 0.5.0, when the package was called llm-failover? Run pip uninstall llm-failover first.

Quick Start

from llm_failover_chain import LLMFailover
from llm_failover_chain.providers import groq, gemini

# Set GROQ_API_KEY and GEMINI_API_KEY as env vars, or pass directly
failover = LLMFailover([
    ("Groq", groq()),
    ("Gemini", gemini()),
])

result = failover.generate("Explain quantum computing in one sentence")
print(result)

Bring Your Own Provider

Any function with the signature (prompt: str, max_tokens: int) -> str works:

import openai

def my_openai(prompt, max_tokens=4096):
    response = openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=max_tokens,
    )
    return response.choices[0].message.content

failover = LLMFailover([
    ("GPT-4o-mini", my_openai),
    ("Groq", groq()),
])

Named Chains

Define multiple chains for different scenarios:

failover = LLMFailover(
    providers=[
        ("Groq", groq()),
        ("Gemini", gemini()),
        ("SambaNova", sambanova()),
    ],
    chains={
        "fast": ["Groq", "Gemini"],             # speed priority
        "reliable": ["Gemini", "Groq"],         # quality priority
        "default": ["Groq", "Gemini", "SambaNova"],
    },
)

# Use a specific chain
result = failover.generate("Quick math: 2+2?", chain="fast")

Per-Provider Budgets

Voluntarily rotate providers before hitting rate limits. After exhausting a provider's budget, the failover engine skips to the next one. Over-budget providers are retried as a last resort before total failure.

failover = LLMFailover(
    providers=[
        ("Groq", groq()),
        ("Gemini", gemini()),
        ("SambaNova", sambanova()),
        ("OpenRouter", openrouter()),
    ],
    budgets={
        "Groq": 8,         # Conservative — daily token cap is the real constraint
        "Gemini": 30,       # Flash free tier has the most headroom
        "SambaNova": 6,     # Unreliable 500s — use sparingly
        "OpenRouter": 50,   # Paid final boss — only hit if everything else is down
    },
)

Calls are round-robin rotated across the chain automatically, so load spreads evenly even without budgets.

Validating Answers

An error isn't the only way a provider can fail you: it can answer with something you can't use. Pass accept (a function from the answer text to True/False) and a rejected answer fails over to the next provider exactly like an error does. Empty answers always fail over.

import json

def is_json_list(text):
    try:
        return isinstance(json.loads(text), list)
    except ValueError:
        return False

ideas = failover.generate("Three product names as a JSON list of strings", accept=is_json_list)

accept should not raise; an exception from it propagates to the caller.

Blackout Callback

Get notified when all providers fail:

import requests

def alert_discord(chain, exhausted):
    requests.post(WEBHOOK_URL, json={
        "content": f"LLM blackout — all providers failed (chain={chain}, exhausted: {exhausted})"
    })

failover = LLMFailover(
    providers=[("Groq", groq()), ("Gemini", gemini())],
    on_blackout=alert_discord,
)

The callback fires once per session (not on every failed call). Call failover.reset() to re-arm it.

Built-in Providers

All helpers accept optional api_key and model overrides:

from llm_failover_chain.providers import groq

custom_groq = groq(api_key="sk-...", model="llama-3.1-8b-instant")

How It Works

1. Round-robin rotate the chain so calls spread across providers
2. Try the next provider in the rotated chain (skip if over budget)
3. If it fails with a rate-limit error (429), retry with exponential backoff; if it fails with a payment error (402), skip it for the session without retrying
4. If it hits a daily quota, mark it as exhausted and skip it for the session
5. If it fails for any other reason, move to the next provider
6. An empty answer, or one accept rejects, counts as a failure and moves on to the next provider
7. If all in-budget providers fail, retry over-budget providers as a last resort
8. If everything fails, fire the blackout callback and raise RuntimeError

Call failover.reset() to clear exhausted providers, call counts, and the blackout flag (e.g., at the start of a new day).

Logging

Uses Python's standard logging module under the llm_failover_chain logger:

import logging
logging.basicConfig(level=logging.INFO)

License

MIT

_Last verified by CI: