voxroutersho 0.1.0

Drop-in hybrid local/remote LLM routing for token-efficient AI apps

voxrouter

Drop-in hybrid local/remote LLM routing. Send every prompt to the cheapest model that can actually handle it — free local models for simple tasks, remote models only when needed.

pip install voxrouter

Quick start

import asyncio
from voxrouter import route

async def main():
    result = await route("What is the capital of France?")
    print(result.answer)         # "Paris"
    print(result.route)          # "local"
    print(result.cost_usd)       # 0.0

asyncio.run(main())

Complex prompts automatically go remote:

result = await route(
    "Design a rate limiter for an API gateway handling 50,000 req/s",
    gemini_api_key="your-key-here",
)
print(result.route)              # "remote"
print(result.model_used)         # "gemini/gemini-2.5-flash-lite"

Setup

VoxRouter needs two things to actually run tasks:

1. A local model via Ollama for cheap/simple tasks:

   ollama serve
   ollama pull llama3.2:1b
   ollama pull qwen2.5:3b
   ollama pull phi3.5:3.8b
   

2. A Gemini API key for complex tasks — get one free at aistudio.google.com/apikey:

   export GEMINI_API_KEY=your-key-here
   

Without Ollama running, local-routed calls will raise ConnectionError. Without a Gemini key, remote-routed calls will raise ValueError. Both are intentional — see Error handling below.

Class-based usage

For repeated calls with the same configuration:

from voxrouter import VoxRouter

vr = VoxRouter(gemini_api_key="your-key", ollama_host="http://localhost:11434")

result1 = await vr.route("Is 17 a prime number?")
result2 = await vr.route("Explain how transformers use attention")

Forcing a route

# Skip escalation — answer locally no matter what
result = await route(prompt, force_local=True)

# Skip local entirely — always use the remote model
result = await route(prompt, force_remote=True)

Task type hints

Giving a hint improves routing accuracy for ambiguous prompts:

result = await route(
    "What's the time complexity here?",
    task_type="factual",   # nudges toward local
)

Valid values: factual, boolean, classification, extraction, reasoning, generation, code, math_proof.

How routing works

1. The prompt is classified into a complexity tier (1–5) using pattern matching, structural analysis, and vocabulary entropy.
2. Tiers 1–2 go to a local Ollama model. Tiers 3–5 go to Gemini.
3. If a local answer comes back with confidence below confidence_threshold (default 0.72), it's automatically escalated to remote — you get the better answer without needing to detect the failure yourself.

RouteResult fields

result.answer                # the model's response
result.route                 # "local" | "remote"
result.model_used            # e.g. "local/qwen2.5:3b" or "gemini/gemini-2.5-flash-lite"
result.complexity_score      # 1-5
result.complexity_label      # "trivial" | "simple" | "moderate" | "complex" | "expert"
result.tokens_used           # total tokens for this call
result.cost_usd              # 0.0 for local, real cost for remote
result.latency_ms            # end-to-end latency
result.confidence            # 0.0-1.0
result.escalated             # True if it started local and moved to remote
result.escalation_reason     # why, if escalated

Error handling

try:
    result = await route(prompt, force_local=True)
except ConnectionError:
    print("Ollama isn't running — start it with `ollama serve`")

try:
    result = await route(prompt, force_remote=True)
except ValueError:
    print("No Gemini API key configured")

License

MIT — see LICENSE. If VoxRouter saves you money in production, a star or a mention is appreciated but never required.