llm-rotate

llm-rotate is a Python library for resilient LLM calls across providers. It provides API key rotation, health-aware selection, retry/failover, and a unified chat interface.

Built by Research Commons.

What it does

• Unified API for OpenAI, Anthropic, Google AI Studio, Vertex AI, and OpenRouter
• Automatic key rotation on rate-limit/auth/transient failures
• Provider fallback chains
• Streaming and sync/async chat support
• Structured per-call usage logging

Quickstart

llm-rotate ships a built-in provider catalog covering all five supported backends. Configure it programmatically with a plain Python dict before your first lm call — no env var registry blob required.

from llm_rotate import configure, lm

configure(
    registry={
        "keys": [
            {
                "key_id": "openai-1",
                "provider": "openai",
                "secret_ref": "env://OPENAI_API_KEY",
                "models": ["gpt-4o-mini"],
            },
            {
                "key_id": "gemini-1",
                "provider": "google_ai_studio",
                "secret_ref": "env://GOOGLE_API_KEY",
                "models": ["gemini-2.0-flash"],
            },
        ]
    },
    use_keys=["openai-1", "gemini-1"],
)

response = await lm.chat("gpt-4o-mini", [{"role": "user", "content": "Hello"}])
print(response.content)

configure() must be called before the first use of lm. Using lm without prior configure() raises ConfigurationError.

Required fields per key entry

Overriding built-in provider settings (optional)

If you need custom cooldown/quarantine values for a specific provider, add a providers key containing only the entries you want to override — the rest keep their built-in defaults:

configure(
    registry={
        "providers": {
            "openai": {
                "provider_type": "direct",
                "display_name": "OpenAI",
                "default_cooldown_seconds": 10,
            }
        },
        "keys": [...],
    },
    use_keys=[...],
)

Provider fallback chains

When all keys for a model's native provider are exhausted (rate-limited, quarantined, or auth-failed), llm-rotate can transparently re-try the same request against another provider. Configure named chains via fallback_chains on configure():

configure(
    registry={
        "keys": [
            {"key_id": "anthropic-1", "provider": "anthropic",
             "secret_ref": "env://ANTHROPIC_API_KEY",
             "models": ["claude-haiku-4-5-20251001"]},
            {"key_id": "openrouter-1", "provider": "openrouter",
             "secret_ref": "env://OPENROUTER_API_KEY",
             "models": ["anthropic/claude-haiku-4.5"]},
        ]
    },
    use_keys=["anthropic-1", "openrouter-1"],
    fallback_chains={
        "anthropic": [{"provider": "openrouter"}],
        "openai":    [{"provider": "openrouter"}],
    },
)

# No explicit `provider=` → provider is inferred from the model, then the
# chain registered under that provider name is walked in order on exhaustion.
response = await lm.chat(
    "claude-haiku-4-5-20251001",
    [{"role": "user", "content": "Hello"}],
)

Resolution rules:

• Explicit provider= argument → only that provider's keys are tried; the chain is not consulted.
• No provider= → provider is inferred from the model name; on exhaustion the chain keyed by that provider name is walked in order.
• No provider= and no chain configured for the inferred provider → ConfigurationError.
• Each entry is {"provider": "<name>", "upstream": "<optional>"}; upstream is used by aggregators like OpenRouter to pin a backend.
• Model-ID compatibility on the fallback provider is the caller's responsibility (e.g. OpenRouter uses anthropic/claude-haiku-4.5, not the native Anthropic model ID).

Merge rules

1. Built-in providers (openai, anthropic, google_ai_studio, google_vertex, openrouter) are always the base layer.
2. Any provider entry in the registry dict's "providers" block replaces the corresponding built-in entry by name; providers not mentioned keep their built-in values.
3. "keys" always come from the caller-supplied dict — the package ships no keys, credentials, or secret_ref values.