smartrouter 1.2.6

智能模型路由网关 — 基于 LiteLLM 的多服务商自动路由 CLI 工具

English | 中文

Smart Router — Intelligent Model Routing Gateway

A multi-provider model intelligent routing CLI tool based on LiteLLM. Exposes a unified OpenAI API interface and automatically selects the most suitable underlying LLM based on task type and difficulty.

Features

• 🔑 Single Entry: One API Key manages all providers
• 🧠 Smart Routing: Auto-detects task types (coding/writing/reasoning/...) and selects optimal models
• 🏷️ Stage Markers: Explicit routing control with [stage:code_review]
• 🔄 Auto Fallback: Automatic model upgrade and retry on failure
• 🌐 Multi-Provider: Supports OpenAI, Anthropic, Qwen, Kimi, MiniMax, GLM, etc.
• 📊 Web Dashboard: Built-in dashboard for real-time monitoring, analytics, and model management

---

📸 Screenshots

Dashboard — service status, model error stats, and quick routing test

Data Analysis — cost trends, token consumption, and request analytics

Model Usage — distribution, ranking, and recent routing records

Model Management — multi-provider configuration and capability overview

Real-time Logs — live streaming with level-based filtering

CLI — intuitive command-line interface with full command reference

---

🚀 5-Minute Quick Start

Prerequisites

• Python 3.9+ is required

1. Installation

Choose one of the following methods:

Option A: pip install (Recommended)

pip install smartRouter

Option B: One-line curl install

curl -fsSL https://raw.githubusercontent.com/vaycentsun/smartRouter/main/script/install-remote.sh | bash

Option C: Homebrew (macOS/Linux)

brew tap vaycentsun/smart-router https://github.com/vaycentsun/smartRouter.git
brew install smart-router

Option D: Local install (from source)

git clone https://github.com/vaycentsun/smartRouter.git
cd smartRouter
./script/install.sh

Option E: Docker (No Python/Node.js needed)

The fastest way to get started without installing any dependencies:

# 1. Create config directory
mkdir -p config

# 2. Download example configs
curl -sSL https://raw.githubusercontent.com/vaycentsun/smartRouter/main/config/examples/v3/providers.yaml -o config/providers.yaml
curl -sSL https://raw.githubusercontent.com/vaycentsun/smartRouter/main/config/examples/v3/models.yaml -o config/models.yaml
curl -sSL https://raw.githubusercontent.com/vaycentsun/smartRouter/main/config/examples/v3/routing.yaml -o config/routing.yaml

# 3. Start with docker-compose
docker-compose up -d

Or use Docker directly:

docker run -d \
  --name smart-router \
  -p 4000:4000 \
  -p 8080:8080 \
  -e SMART_ROUTER_MASTER_KEY="your-master-key" \
  -e OPENAI_API_KEY="sk-..." \
  -v "$(pwd)/config:/app/config:ro" \
  your-dockerhub-username/smartrouter:latest

Note: Replace your-dockerhub-username with your actual Docker Hub username, or build locally with docker build -t smartrouter .

Ports: 4000 for Proxy API, 8080 for Web Dashboard.

See Docker Guide for full details including image tags, multi-arch support, and configuration.

Uninstall

One-line uninstall:

curl -fsSL https://raw.githubusercontent.com/vaycentsun/smartRouter/main/script/uninstall.sh | bash

Or manually:

# Stop service
smart-router stop

# Uninstall package
pip uninstall smartRouter

# Clean up data
rm -rf ~/.smart-router

2. Initialize Configuration

# Download config files via curl (no pip install needed)
curl -sSL https://raw.githubusercontent.com/vaycentsun/smartRouter/main/script/download-config.py | python3

# Or via CLI after pip install
smart-router init

# Force overwrite existing files
smart-router init --force

# Specify custom directory
smart-router init --output ./my-config

Edit the three config files:

vim ~/.smart-router/providers.yaml  # API keys and base URLs
vim ~/.smart-router/models.yaml     # Model capabilities
vim ~/.smart-router/routing.yaml    # Task definitions and routing strategies

Smart Router uses a three-file decoupled architecture:

• providers.yaml - API keys and base URLs per provider
• models.yaml - Model capabilities (quality/cost scores)
• routing.yaml - Task definitions and routing strategies

See Configuration Guide for details.

3. Start Service (Background)

# Start in background (recommended)
smart-router start

# Check status
smart-router status

# Output example:
# ● Smart Router is running
#   PID: 12345
#   Service: http://127.0.0.1:4000
#   Logs: ~/.smart-router/smart-router.log

Foreground mode (for debugging):

export OPENAI_API_KEY="your-key"
smart-router start --foreground

4. Test Routing (No Model Call)

# Test auto-routing
smart-router dry-run "Review this Python code"

# Use stage marker
smart-router dry-run "[stage:writing] Write a business email" --strategy cost

5. Client Usage

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:4000",
    api_key="sk-smart-router-local"
)

# Auto-routing
response = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Review this code"}]
)

# Use stage marker
response = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "[stage:code_review] Review this code"}]
)

---

📋 Quick Command Reference

Service Management

Command	Description
smart-router start	Start service in background
smart-router start --foreground	Start in foreground (debug mode)
smart-router stop	Stop service
smart-router restart	Restart service
smart-router status	Check service status

Logs

Command	Description
smart-router logs	View last 50 lines of logs
smart-router logs -n 100	View last 100 lines
smart-router logs -f	Follow logs (Ctrl+C to exit)

Configuration & Testing

Command	Description
smart-router init	Generate default configuration

| smart-router doctor | Run health check (includes config validation) | | smart-router dry-run "prompt text" | Test routing decision | | smart-router list | List configured providers and models |

---

🎯 Stage Markers

Add markers to prompts for explicit routing control:

# Code review
"[stage:code_review] Review this code"

# Writing task (easy)
"[stage:writing] [difficulty:easy] Write an email"

# Complex reasoning
"[stage:reasoning] [difficulty:hard] Prove this theorem"

Supported Stages

Marker	Purpose	Default Models
brainstorming	Brainstorming ideas	qwen-turbo, gpt-4o-mini
code_review	Code review	claude-3-sonnet
writing	Writing tasks	qwen-turbo, kimi-k2
reasoning	Logical reasoning	claude-3-opus
chat	General chat	qwen-turbo, gpt-4o-mini

More details: Stage Marker System

---

⚙️ Configuration

Smart Router uses a three-file decoupled architecture:

config/
├── providers.yaml    # Provider connection settings
├── models.yaml       # Model capability declarations  
└── routing.yaml      # Task definitions and routing rules

providers.yaml

Define API endpoints and authentication once per provider:

providers:
  openai:
    api_base: https://api.openai.com/v1
    api_key: os.environ/OPENAI_API_KEY
    timeout: 30
    
  anthropic:
    api_base: https://api.anthropic.com
    api_key: os.environ/ANTHROPIC_API_KEY

models.yaml

Declare model capabilities (quality/cost scores 1-10):

models:
  gpt-4o:
    provider: openai              # References provider above
    litellm_model: openai/gpt-4o
    capabilities:
      quality: 9                  # Quality score (1-10)
      cost: 3                     # Cost efficiency (10=cheapest)
      context: 128000             # Context window
    supported_tasks: [chat, code_review, writing]
    difficulty_support: [easy, medium, hard]

routing.yaml

Define tasks and routing strategies:

tasks:
  code_review:
    name: "Code Review"
    description: "Review code quality"
    capability_weights:           # How to weight capabilities
      quality: 0.6                # 60% weight on quality
      cost: 0.4                   # 40% weight on cost

strategies:
  auto:     # Uses task weights to calculate composite score
  cost:     # Selects cheapest model (with quality threshold filter)

fallback:
  mode: auto
  similarity_threshold: 2  # Models within ±2 quality are fallbacks

Key advantages:

• Add/remove models by editing only models.yaml
• Routing is dynamically calculated - no manual list maintenance
• Fallback chains auto-derived from capability similarity
• Clear separation of concerns

Routing Strategies

• auto: Automatically calculate best model based on task weights
• cost: Select cheapest model (with quality threshold filter)

More configuration details: Configuration Guide

---

🔧 Troubleshooting

Service Won't Start

# Check port usage
lsof -i :4000

# View logs
smart-router logs

# Run health check (includes config validation)
smart-router doctor

# Run in foreground to see detailed errors
smart-router start --foreground

Check Environment Variables

# Check API Keys
echo $OPENAI_API_KEY
echo $ANTHROPIC_API_KEY

# Set environment variable
export OPENAI_API_KEY="sk-..."

Test Connectivity

# Test if service is running
curl http://localhost:4000/v1/models \
  -H "Authorization: Bearer sk-smart-router-local"

More troubleshooting: Troubleshooting Guide

---

📚 Documentation

Document	Content
Complete Guide	Detailed CLI commands, configuration, best practices
V3 Config Examples	V3 three-file configuration examples
V3 Design Spec	V3 architecture design and migration guide
Design Doc	Original architecture design and technical specs

Quick Navigation

• CLI Commands - Complete guide for start, stop, dry-run, validate
• Client Integration - Python, JavaScript, Cursor, Claude Code setup
• Best Practices - Configuration management, cost optimization
• Advanced Usage - Custom task types, Fallback chain config
• Troubleshooting - Detailed troubleshooting steps

---

🏗️ Architecture

User Request → LiteLLM Proxy → SmartRouter Plugin
                                  ├── Stage Marker Parsing
                                  ├── Task Classification (L1 Rules + L2 Similarity)
                                  ├── Model Selection (auto/cost)
                                  └── Fallback Management
                    ↓
            Target Model Provider

Components

• core/smart_router/cli.py - CLI entry commands
• core/smart_router/plugin.py - SmartRouter core plugin (V2 config)
• core/smart_router/plugin_v3_adapter.py - V3 configuration adapter
• core/smart_router/server.py - LiteLLM Proxy wrapper
• core/smart_router/config/ - Configuration loading and validation
  • v3_schema.py - V3 Pydantic schemas
  • v3_loader.py - V3 three-file config loader
• core/smart_router/classifier/ - Task classifier (L1 Rules + L2 Embedding)
• core/smart_router/selector/ - Model selection strategies
  • v3_selector.py - V3 capability-based selector
• core/smart_router/utils/ - Utility functions

---

🧪 Development

pip install -e ".[dev]"

pytest tests/ -v

---

📄 License

MIT