routeme 0.1.2

Intelligent LLM Router - Route requests across multiple LLM providers with automatic failover, quota management, and caching

Intelligent LLM Router

Intelligent LLM Router is a Python library that routes requests across multiple LLM providers with automatic failover, quota management, and response caching.

Features

• Multi-Provider Support: OpenAI, Anthropic, Google Gemini, Groq, Mistral, Cohere, DeepSeek, Together, HuggingFace, OpenRouter, xAI, DashScope, and Ollama
• Automatic Failover: Automatically switches to the next provider if one fails
• Quota Management: Tracks RPM/RPD limits per provider and routes around rate limits
• Response Caching: Exact and semantic caching to reduce costs and latency
• Multiple Routing Strategies: Auto, Cost-Optimized, Quality-First, Latency-First, Round-Robin
• Vision & Embeddings: Full support for vision models and embeddings

Installation

pip install llm_router

Or with specific extras:

pip install llm_router[server]  # Includes FastAPI server

Quick Start

import asyncio
from llm_router import IntelligentRouter, RoutingOptions, RoutingStrategy

async def main():
    # Initialize the router
    router = IntelligentRouter()
    await router.start()
    
    # Define your request
    request_data = {
        "messages": [
            {"role": "user", "content": "What is the capital of France?"}
        ],
        "temperature": 0.7,
    }
    
    # Configure routing options (optional)
    options = RoutingOptions(
        strategy=RoutingStrategy.AUTO,
    )
    
    # Route the request
    response = await router.route(request_data, options)
    
    print(response["choices"][0]["message"]["content"])
    print(f"Provider: {response['routing_metadata']['provider']}")
    
    await router.stop()

asyncio.run(main())

Configuration

Environment Variables

All configuration can be done via environment variables with the ROUTER_ prefix:

Variable	Default	Description
ROUTER_LLM_TIMEOUT	60	Timeout for LLM calls in seconds
ROUTER_MAX_RETRIES	3	Maximum retry attempts
ROUTER_ENABLE_OLLAMA_FALLBACK	true	Enable Ollama fallback when cloud providers fail
ROUTER_CACHE_DIR	/tmp/llm_router_cache	Directory for response cache
ROUTER_RESPONSE_CACHE_TTL	3600	Cache TTL in seconds
ROUTER_DEFAULT_STRATEGY	auto	Default routing strategy

Provider API Keys

Set API keys as environment variables:

export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
export GROQ_API_KEY=gsk_...
export GEMINI_API_KEY=AIza...
# etc.

Routing Strategies

Strategy	Description
auto	Balanced: quota + latency + quality (default)
cost_optimized	Maximize remaining free quota
quality_first	Prioritize highest quality models
latency_first	Prioritize fastest responding models
round_robins	Uniform spread across providers

Request Options

from llm_router import RoutingOptions, CachePolicy

options = RoutingOptions(
    strategy=RoutingStrategy.COST_OPTIMIZED,
    free_tier_only=True,  # Only use free tier providers
    preferred_providers=["groq", "gemini"],  # Prefer these providers
    excluded_providers=["openai"],  # Skip these providers
    cache_policy=CachePolicy.ENABLED,  # Enable response caching
)

API Reference

IntelligentRouter

router = IntelligentRouter()
await router.start()  # Initialize router
response = await router.route(request_data, options)  # Route request
stats = router.get_stats()  # Get router statistics
await router.stop()  # Cleanup

Models

• RoutingOptions: Configuration for routing behavior
• RoutingStrategy: Enum for routing strategies
• TaskType: Enum for task types (chat, embeddings, vision, etc.)
• CachePolicy: Enum for caching behavior
• settings: Global settings object

Running the Server

# Install with server extras
pip install llm_router[server]

# Run the server
llm-router

# Or with custom settings
ROUTER_PORT=8000 llm-router

The server provides a FastAPI-compatible API at /v1/chat/completions, /v1/embeddings, and /health.

Development

# Clone the repository
git clone https://github.com/anomalyco/llm_router.git

# Install in development mode
pip install -e ".[dev]"

# Run tests
pytest

# Run linting
ruff check src/

License

MIT License - see LICENSE for details.