oq-ai-router

Provider-agnostic LLM routing for Python. Free-first fallback chains, a routing policy supplied by your application, and per-tenant credentials — with the actual transport delegated to LiteLLM.

Extracted from a production application rather than designed in the abstract.

What this is, and what it deliberately is not

The hard-won lesson behind this package: most of an "AI layer" is already solved, and the part that isn't is the part specific to your application.

So this is a thin policy layer, not a gateway and not a framework. It ships no routing table of its own — a library that hardcodes one app's taxonomy is not reusable, and adopting it would mean editing the library.

Install

pip install oq-ai-router

Use

Define your application's own categories and model preferences, register the policy once at startup, then call:

import oq_ai_router
from oq_ai_router import AiConfig, TablePolicy, complete_json

MY_POLICY = TablePolicy(
    # Free sources first, paid last. This ORDER is the free-first mechanism.
    provider_order=["omniroute", "openrouter", "groq", "anthropic"],
    models={
        "omniroute":  {"chat": ["auto/offline", "auto/cheap"]},
        "openrouter": {"chat": ["openrouter/free"]},
        "groq":       {"chat": ["llama-3.3-70b-versatile"]},
        "anthropic":  {"chat": ["claude-3-5-haiku-latest"]},
    },
)
oq_ai_router.set_default_policy(MY_POLICY)

result = complete_json(
    "chat",
    [{"role": "user", "content": "Reply as JSON: {\"ok\": true}"}],
    config=AiConfig(keys={"groq": "gsk-..."}),
)
result["data"]        # the parsed object
result["model_used"]  # which model actually answered

Categories are yours. One app's are explain | suggest | audit | concierge, another's are vision | json | prose; both route through this package with no edits to it. If table-driven ordering cannot express what you need (latency budgets, freshness windows, live quota telemetry), implement the two-method RoutingPolicy protocol directly.

Per-tenant keys

AiConfig is injected by the host — the package never reads env, a database or app settings on its own. Multi-tenant hosts pass a different AiConfig per request; env_config() is the zero-config path for single-tenant use.

Zero-credential operation

Providers may declare requires_key=False for a local gateway that holds the upstream credentials itself. The bundled omniroute entry does this, so a fresh self-hoster running OmniRoute gets a working AI chain with no API keys at all.

Verified 2026-07-24: a completion through oq_ai_router → LiteLLM Router → a keyless OmniRoute returned successfully.

Do not attempt to route a consumer AI subscription (Claude Pro/Max, ChatGPT Plus, Gemini One) through this or any proxy. Those OAuth flows are scoped to each vendor's own first-party clients; reusing them breaches the consumer terms and risks the account. Use metered API keys (every major provider has a free tier) or a genuinely free lane.

Design notes worth knowing

• A Router is cached per (chain, credentials). LiteLLM's Router expects a static deployment list, but per-tenant keys resolve per request. Cache keys are hashed, so a credential cannot leak through a dict repr or a traceback.
• Cooldowns are per Router, therefore shared only between tenants on the same credentials — which is the correct boundary, since separate keys have separate quotas.
• Per-provider timeouts. A direct paid API should fail fast so the chain moves on; a local gateway draining free quota across upstreams legitimately takes longer. Measured, not guessed.
• JSON validity is handled here, not by LiteLLM, which has no concept of "returned 200, body would not parse". Free models emit think-tags, code fences and trailing commas; parse_lenient copes, and unparseable output falls through to the next model.

Evals and cost guardrails

Model ids rotate and free models vary wildly, so "which model for which category" is an empirical question. oq_ai_router.evals answers it by measurement: run a corpus across every model in a chain and take the cheapest model that still passes — accuracy, cost and latency scored together.

Two corpora, both first-class. A clean synthesized corpus measures the happy path; the failures that actually matter arrive wearing real client data that can never be committed. So:

Both run through one schema and one scorer, and results are reported separately: a green public score beside a red private score means the contract holds but reality has moved, which is the most actionable signal here. Private cases never expose their text or id in a report — only the failure kind.

Closing the loop is the point. When a private case fails, derive a synthesized twin with synthesize_case, read it yourself, and commit it to the public corpus. Over time the public corpus accumulates the shape of every real failure without absorbing a byte of real data. Redaction handles the identifier formats it knows (PAN, GSTIN, CIN, email, phone) and is a starting point for human review, never a security boundary.

oq_ai_router.budget adds the spend ceilings: a per-request cap (pushed upstream as X-OmniRoute-Budget where the gateway enforces it) and a per-tenant ledger for the slow bleed no single request would trip. Cost is recorded even when a cap is breached — money spent is spent, and dropping it would let violations repeat indefinitely.

Licence

Dual licensed: AGPL v3 or later for community use, plus a proprietary commercial licence for anyone who needs to keep modifications private or embed this in a closed-source product. See LICENSING.md.

Note the Affero clause: running a modified version as a network service obliges you to offer its source to that service's users.

A CLA must be in place before any outside contribution is accepted, because dual licensing only works while the copyright is wholly owned. Until then this repository does not accept contributions.