llm-rotate 0.3.0

Unified LLM client with key rotation, health-aware failover, and multi-provider orchestration.

llm-rotate

llm-rotate is a Python library for resilient LLM calls across providers. It provides API key rotation, health-aware selection, retry/failover, and a unified chat interface.

What it does

• Unified API for OpenAI, Anthropic, Google AI Studio, Vertex AI, and OpenRouter
• Automatic key rotation on rate-limit/auth/transient failures
• Provider fallback chains
• Streaming and sync/async chat support
• Structured per-call usage logging

Quickstart

llm-rotate ships a built-in provider catalog covering all five supported backends. Configure it programmatically with a plain Python dict before your first lm call — no env var registry blob required.

from llm_rotate import configure, lm

configure(
    registry={
        "keys": [
            {
                "key_id": "openai-1",
                "provider": "openai",
                "secret_ref": "env://OPENAI_API_KEY",
                "models": ["gpt-4o-mini"],
            },
            {
                "key_id": "gemini-1",
                "provider": "google_ai_studio",
                "secret_ref": "env://GOOGLE_API_KEY",
                "models": ["gemini-2.0-flash"],
            },
        ]
    },
    use_keys=["openai-1", "gemini-1"],
)

response = await lm.chat("gpt-4o-mini", [{"role": "user", "content": "Hello"}])
print(response.content)

configure() must be called before the first use of lm. Using lm without prior configure() raises ConfigurationError.

Required fields per key entry

Field	Description
key_id	Unique name for this credential (used in logs and use_keys=[...])
provider	One of: openai, anthropic, google_ai_studio, google_vertex, openrouter
secret_ref	env://VAR_NAME — resolved at runtime, never stored
models	List of model IDs this key may serve (used for routing and inference)

Overriding built-in provider settings (optional)

If you need custom cooldown/quarantine values for a specific provider, add a providers key containing only the entries you want to override — the rest keep their built-in defaults:

configure(
    registry={
        "providers": {
            "openai": {
                "provider_type": "direct",
                "display_name": "OpenAI",
                "default_cooldown_seconds": 10,
            }
        },
        "keys": [...],
    },
    use_keys=[...],
)

Merge rules

1. Built-in providers (openai, anthropic, google_ai_studio, google_vertex, openrouter) are always the base layer.
2. Any provider entry in the registry dict's "providers" block replaces the corresponding built-in entry by name; providers not mentioned keep their built-in values.
3. "keys" always come from the caller-supplied dict — the package ships no keys, credentials, or secret_ref values.