llm-route

Intelligent LLM routing library for Python. Drop-in replacement for direct Azure OpenAI calls with automatic load balancing, 429 failover, and request class isolation.

Built as a minimal-dependency alternative to LiteLLM Router, focused on security and transparency.

Features

• Weighted least-outstanding routing — routes to the deployment with the lowest load relative to its capacity
• 429-aware cooldown — respects Retry-After headers, automatically fails over to the next backend
• Request class isolation — separate concurrency budgets for light/medium/heavy requests prevent expensive operations from starving fast ones
• Token-aware capacity tracking — tracks actual token usage per deployment per minute window
• Request deadline — enforces a total timeout across all retry attempts
• Health reporting — exposes deployment health, inflight counts, 429 rates, and TPM usage

Install

pip install llm-route

Or with uv:

uv add llm-route

Quick Start

import asyncio
from llm_route import SmartRouter, RequestClass, RouterConfig
from llm_route.config import DeploymentConfig

config = RouterConfig(
    deployments=[
        DeploymentConfig(
            name="eastus-1",
            endpoint="https://my-resource.openai.azure.com/",
            api_key="your-key",
            deployment_name="gpt-4.1",
            tpm_quota=120_000,
        ),
    ],
)

router = SmartRouter(config=config)

async def main():
    response = await router.complete(
        messages=[{"role": "user", "content": "Hello"}],
        request_class=RequestClass.LIGHT,
    )
    print(response.choices[0].message.content)

asyncio.run(main())

Configuration

JSON config file

{
  "deployments": [
    {
      "name": "eastus-1",
      "endpoint": "https://my-eastus-1.openai.azure.com/",
      "api_key": "your-api-key",
      "deployment_name": "gpt-4.1",
      "tpm_quota": 120000
    },
    {
      "name": "eastus-2",
      "endpoint": "https://my-eastus-2.openai.azure.com/",
      "api_key": "your-api-key",
      "deployment_name": "gpt-4.1",
      "tpm_quota": 60000
    }
  ],
  "default_timeout": 60.0,
  "max_retries": 3,
  "cooldown_seconds": 10.0,
  "concurrency": {
    "light": 20,
    "medium": 10,
    "heavy": 3
  }
}

Load it:

config = RouterConfig.from_file("config.json")
router = SmartRouter(config=config)

Environment variables

All settings can be set via env vars with LLM_ROUTE_ prefix:

LLM_ROUTE_DEFAULT_TIMEOUT=60.0
LLM_ROUTE_MAX_RETRIES=3
LLM_ROUTE_COOLDOWN_SECONDS=10.0

Model compatibility (2026-08)

Providers reject different parameters per model family. The router adapts requests automatically and reports what it changed.

Azure/OpenAI reasoning models — deployment names starting with o1, o3, o4, or gpt-5 (case-insensitive) are auto-detected as reasoning models: max_tokens is translated to max_completion_tokens, and temperature / top_p / presence_penalty / frequency_penalty are stripped. gpt-4o, gpt-4.1, and the chat-tuned gpt-5-chat / gpt-5-chat-latest are not reasoning models. Set reasoning_model: true or false on the deployment to override the auto-detection — needed when the Azure deployment name doesn't match the underlying model name (e.g. a deployment called prod-fast serving gpt-5-mini).

Claude 5-family sampling params — claude-opus-4-7, claude-opus-4-8, claude-opus-5, claude-sonnet-5, claude-fable-5, and claude-mythos models return HTTP 400 for any explicit temperature / top_p / top_k, so the router strips them. Set supports_sampling: true or false on the Anthropic deployment to override.

The two Anthropic flags are different axes — on an Anthropic deployment, supports_sampling governs REQUEST BEHAVIOUR (whether temperature / top_p / top_k are sent, stripped, or raise), while reasoning_model is REPORTING ONLY (it sets is_reasoning, which drives reasoning_deployments in the health report and the startup family checks) and never changes what is sent to the API. Set both if a deployment needs both.

Seeing what was changed — stripped parameters are reported on the router result (sanitized_params) and logged. Set strict_param_validation: true in the router config to raise ReasoningModelParamError instead of silently sanitizing: the Anthropic provider raises for ANY parameter it would otherwise strip or drop — sampling params on a rejecting model, params with no Messages API equivalent (response_format, seed, logprobs, top_logprobs, n, presence_penalty, frequency_penalty), and untranslated params (tools, tool_choice, thinking, ...). The Azure provider raises for reasoning-model sampling params.

Claude refusals — a safety-classifier refusal returns HTTP 200 with empty content and finish_reason == "content_filter". Callers that assume a successful response has text should check finish_reason before using choices[0].message.content.

finish_reason == "length" has two causes — the max_tokens output cap (stop_reason: "max_tokens") and context-window exhaustion (stop_reason: "model_context_window_exceeded", logged as anthropic_context_window_exceeded). Don't blindly retry with a larger max_tokens: that fixes the first cause and makes the second worse. Shorten the input instead.

Request Classes

Request classes provide concurrency isolation per deployment. Heavy requests (full document review) won't starve light ones (quick text search).

await router.complete(
    messages=[...],
    request_class=RequestClass.HEAVY,  # uses the heavy concurrency budget
)

How Routing Works

1. Filter — remove disabled, cooled-down, and already-tried deployments
2. Filter — remove deployments with no available concurrency for the request class
3. Score — inflight / weight where weight = tpm_quota / min_tpm. Lower is better.
4. Select — pick lowest score; break ties by remaining TPM headroom
5. Execute — acquire semaphore, call Azure OpenAI with remaining deadline
6. Failover — on 429 or 5xx, mark cooldown, try next deployment
7. Deadline — if total timeout expires, raise RouterExhaustedError

Health Monitoring

health = router.health()
for dep in health.deployments:
    print(f"{dep.name}: healthy={dep.healthy}, inflight={dep.inflight}, "
          f"tpm={dep.tpm_used}/{dep.tpm_quota}, 429s={dep.total_429s}")

Expose as a FastAPI endpoint:

@app.get("/health/llm")
async def llm_health():
    return router.health().model_dump()

Dependencies

Minimal by design:

• openai — Azure OpenAI SDK
• pydantic — data validation
• pydantic-settings — configuration management

Optional:

• redis — reserved for planned shared-state support (pip install llm-route[redis]). Not yet implemented: cooldown/TPM/concurrency state is currently per-process (the redis_url config field is accepted but unused). Multi-replica or multi-app deployments sharing the same backends will each track 429s independently until this lands.

Security

This library was built in response to the LiteLLM supply chain attack (March 2026). Design principles:

• Minimal dependencies — 3 required packages, all well-maintained
• No build-time code execution — pure Python, no compiled extensions
• uv.lock committed — exact dependency tree is auditable
• OIDC publishing — no stored PyPI tokens in CI

License

MIT