llm-route 0.5.0

Intelligent LLM routing with weighted least-outstanding selection, 429 cooldown, and request class isolation

llm-route

Intelligent LLM routing library for Python. Drop-in replacement for direct Azure OpenAI calls with automatic load balancing, 429 failover, and request class isolation.

Built as a minimal-dependency alternative to LiteLLM Router, focused on security and transparency.

Features

• Weighted least-outstanding routing — routes to the deployment with the lowest load relative to its capacity
• 429-aware cooldown — respects Retry-After headers, automatically fails over to the next backend
• Request class isolation — separate concurrency budgets for light/medium/heavy requests prevent expensive operations from starving fast ones
• Token-aware capacity tracking — tracks actual token usage per deployment per minute window
• Request deadline — enforces a total timeout across all retry attempts
• Health reporting — exposes deployment health, inflight counts, 429 rates, and TPM usage

Install

pip install llm-route

Or with uv:

uv add llm-route

Quick Start

import asyncio
from llm_route import SmartRouter, RequestClass, RouterConfig
from llm_route.config import DeploymentConfig

config = RouterConfig(
    deployments=[
        DeploymentConfig(
            name="eastus-1",
            endpoint="https://my-resource.openai.azure.com/",
            api_key="your-key",
            deployment_name="gpt-4o",
            tpm_quota=120_000,
        ),
    ],
)

router = SmartRouter(config=config)

async def main():
    response = await router.complete(
        messages=[{"role": "user", "content": "Hello"}],
        request_class=RequestClass.LIGHT,
    )
    print(response.choices[0].message.content)

asyncio.run(main())

Configuration

JSON config file

{
  "deployments": [
    {
      "name": "eastus-1",
      "endpoint": "https://my-eastus-1.openai.azure.com/",
      "api_key": "your-api-key",
      "deployment_name": "gpt-4o",
      "tpm_quota": 120000
    },
    {
      "name": "eastus-2",
      "endpoint": "https://my-eastus-2.openai.azure.com/",
      "api_key": "your-api-key",
      "deployment_name": "gpt-4o",
      "tpm_quota": 60000
    }
  ],
  "default_timeout": 60.0,
  "max_retries": 3,
  "cooldown_seconds": 10.0,
  "concurrency": {
    "light": 20,
    "medium": 10,
    "heavy": 3
  }
}

Load it:

config = RouterConfig.from_file("config.json")
router = SmartRouter(config=config)

Environment variables

All settings can be set via env vars with LLM_ROUTE_ prefix:

LLM_ROUTE_DEFAULT_TIMEOUT=60.0
LLM_ROUTE_MAX_RETRIES=3
LLM_ROUTE_COOLDOWN_SECONDS=10.0

Request Classes

Request classes provide concurrency isolation per deployment. Heavy requests (full document review) won't starve light ones (quick text search).

Class	Default concurrency / deployment	Use case
LIGHT	20	Short extraction, find-text, quick Q&A
MEDIUM	10	Clause analysis, section review
HEAVY	3	Full document review, redline, long synthesis

await router.complete(
    messages=[...],
    request_class=RequestClass.HEAVY,  # uses the heavy concurrency budget
)

How Routing Works

1. Filter — remove disabled, cooled-down, and already-tried deployments
2. Filter — remove deployments with no available concurrency for the request class
3. Score — inflight / weight where weight = tpm_quota / min_tpm. Lower is better.
4. Select — pick lowest score; break ties by remaining TPM headroom
5. Execute — acquire semaphore, call Azure OpenAI with remaining deadline
6. Failover — on 429 or 5xx, mark cooldown, try next deployment
7. Deadline — if total timeout expires, raise RouterExhaustedError

Health Monitoring

health = router.health()
for dep in health.deployments:
    print(f"{dep.name}: healthy={dep.healthy}, inflight={dep.inflight}, "
          f"tpm={dep.tpm_used}/{dep.tpm_quota}, 429s={dep.total_429s}")

Expose as a FastAPI endpoint:

@app.get("/health/llm")
async def llm_health():
    return router.health().model_dump()

Dependencies

Minimal by design:

• openai — Azure OpenAI SDK
• pydantic — data validation
• pydantic-settings — configuration management

Optional:

• redis — shared state for multi-replica deployments (pip install llm-route[redis])

Security

This library was built in response to the LiteLLM supply chain attack (March 2026). Design principles:

• Minimal dependencies — 3 required packages, all well-maintained
• No build-time code execution — pure Python, no compiled extensions
• uv.lock committed — exact dependency tree is auditable
• OIDC publishing — no stored PyPI tokens in CI

License

MIT