fastapi-llm-gateway

A composable FastAPI LLM gateway with unified multi-provider support.

Features:

• 🚀 Multiple Usage Patterns: Use as SDK, integrate into FastAPI apps, or run as standalone service
• 🔌 Unified API: Single interface for OpenAI, Anthropic, Cohere, VolcEngine, and 100+ providers
• 🔄 Smart Routing: Built-in load balancing, failover, and model aliasing
• 💰 Cost Tracking: Automatic cost calculation per request
• 🌍 Zero Config: Works out-of-the-box with environment variables
• 📦 PyPI Ready: Install with a single pip command

Installation

pip install fastapi-llm-gateway

Quick Start

Pattern 1: SDK - Programmatic Access

from fastapi_llm_gateway import GatewayClient

# Option A: Pass API keys directly
client = GatewayClient(api_keys={"openai": "sk-..."})

# Option B: Use environment variables (recommended)
export OPENAI_API_KEY="sk-..."
client = GatewayClient()  # Automatically reads from env vars

# Make requests
response = client.chat(
    messages=[{"role": "user", "content": "Hello!"}],
    provider="openai",
    model="gpt-4o"
)
print(response.content)
print(f"Cost: ${response.cost}")

Pattern 2: FastAPI Integration (Recommended)

from fastapi import FastAPI
from fastapi_llm_gateway import GatewayRouter

app = FastAPI()
app.include_router(GatewayRouter(prefix="/api/llm"))

# Your app now has LLM gateway endpoints:
# POST /api/llm/v1/chat/completions
# GET  /api/llm/models
# GET  /api/llm/health

Pattern 3: Standalone Service

# Set your API keys as environment variables
export OPENAI_API_KEY="sk-..."

# Start the server
fastapi-llm-gateway-serve

# Or with custom options
fastapi-llm-gateway-serve --host 0.0.0.0 --port 8080 --workers 4

Configuration

Environment Variables

Built-in Models

The package includes built-in definitions for common models:

• OpenAI: gpt-4o, gpt-4o-mini, gpt-3.5-turbo
• Anthropic: claude-3-5-sonnet, claude-3-opus
• Cohere: command-r, command-r-plus
• VolcEngine: doubao-pro-32k (requires endpoint_id config)

License

MIT License - see LICENSE for details.