ProvizElekto

Smart LLM model router. Picks the best model for each call based on context size, rate limits, and capabilities — and retries automatically on failure.

Your app → pz.call(step, fn)               → CallResult
           pz.call_litellm(step, messages) → CallResult
                    ↕  (automatic)
           select → LLM call → report → retry on failure
                    ↕
              proviz-server (Rust)
          rate-limit state · catalog

Key difference from LiteLLM fallback: LiteLLM retries after failure. ProvizElekto picks the right model before the call — skipping models that are rate-limited or near their quota, can't fit the context, or lack required capabilities — then retries with the next eligible model automatically.

Two roles depending on the path

In the regular flow, the server is a pure router — it picks the model and returns credentials; your code makes the actual LLM call.

In the synchronous /complete flow, the server is the caller — it selects, calls the provider, and reports, all in one round-trip — so your code needs no litellm or provider SDK.

In the batch flow, the server becomes the caller:

# Regular: YOUR code calls the LLM
Your app → POST /select → ModelCandidate → your code → Mistral/OpenAI/...
                                                ↓
                                        POST /report

# Synchronous: the SERVER calls the provider for you
Your app → POST /complete → server selects + calls provider + reports → {text, usage, cost}

# Batch: the SERVER calls Mistral on your behalf
Worker A ──┐
Worker B ──┤ POST /batch/submit → server accumulates over window_secs
Worker C ──┘
                    ↓ server → POST Mistral /v1/batch/jobs (50% discount)
                    ↓ server polls until complete
Worker A ──┐
Worker B ──┤ GET /batch/result/{id} → response
Worker C ──┘

The batch path pools requests from all workers into a single Mistral job — the only way to qualify for Mistral's 50% batch discount. No individual worker can do this on its own, so the server acts as the aggregation point and makes the Mistral call itself.

Deployment note: when using batch, the server process (including Docker) must have the Mistral API key env vars set. In the regular flow, API keys only need to be present in the caller's environment.

Features

• Context-aware selection - don't waste a 128k model on a 1k prompt
• Proactive quota tracking - sliding-window counters (RPM/TPM/RPD/TPD) plus atomic in-flight reservations; avoids over-booking before any 429 fires
• Provider-anchored windows - every successful call forwards x-ratelimit-remaining-* headers back to the server; the window floor is clamped to provider reality so internal estimates can't drift below what the provider actually sees
• Scored selection - multi-component scoring: fast headroom (RPS/RPM/TPM, 25%), daily budget (RPD/TPD, 20%), quality (20%), cost (15%), latency (10%), traffic balance (10%). Over-quota models stay eligible with lower scores — AllModelsExhausted only fires when every model is in reactive 429 cooldown.
• Tunable weights - override the cost/latency/quality weights above per request (cost_weight/latency_weight/quality_weight on /select and /complete) or once per group (proviz group set-weights); omitting them reproduces the built-in weights exactly, and no model is ever hard-excluded by a weight the way a hard filter would.
• Measured per-step quality - POST /catalog/step-quality lets a caller push a real, task-specific quality score (e.g. a benchmark pass-rate) for a (model, step) pair, checked before the model's hand-curated global quality_score.
• OpenRouter provider routing - when the selected model routes through OpenRouter, the same cost_weight/latency_weight also steers OpenRouter's own upstream-provider choice (provider.sort: "price"|"latency") - the bias applies at both routing levels, not just proviz's own model selection.
• Aggregator auto-sync - OpenRouter, Requesty, Nous Portal and OrcaRouter each aggregate hundreds of models with drifting pricing; their models.json is machine-generated from the provider's own /models endpoint (once at startup, then hourly). proviz providers sync-<name> --dry-run to review the mapping. OrcaRouter's /v1/models omits capability flags and cache pricing, so each newly-seen model is additionally scraped once from its SEO detail page (PROVIZ_ORCAROUTER_ENRICH_MAX per run, fail-soft); its real per-request cost comes back as usage.cost_usd when /complete sends the X-OrcaRouter-Include-Cost header, and it gets reasoning_effort: "minimal" by default since it has no reasoning-off switch.
• Cached-input pricing - providers that keep a warm prompt-cache (DeepSeek, Nous Portal, OpenAI, Anthropic) bill cache-hit input tokens at a steep discount and report the hit count in the response. Model.price_cached_input_per_1m (auto-filled from Nous's pricing.input_cache_read, hand-curated elsewhere) makes cost accounting reflect it; /complete parses usage.prompt_tokens_details.cached_tokens (or DeepSeek's prompt_cache_hit_tokens) and echoes it on the response. None = no distinct rate, cost is byte-identical to before.
• Prompt-cache stickiness - proviz group set-sticky --slug <g> --enabled makes the selector nudge consecutive calls for that group toward the model that served the last one, so a large repeated prompt prefix stays cache-warm. A bounded bonus that yields to heatroom: it can't keep a rate-limited or headroom-drained model winning, so rotation under load is unaffected.
• Traffic shaping - per-brand traffic_weight steers load proportionally across providers in a 5-minute rolling window; under-served brands get a higher score on the traffic component
• Capability filtering - hard requirements for function calling, JSON mode
• Language filtering - restrict selection to models declared to support a given language (ISO 639-1), so you never call a model in the wrong language
• Quality floor - reject models below a quality threshold per step
• Model groups - define named pools of models (e.g. "fast-chat", "coding-tier1") and restrict selection to that pool
• Your keys, your models - curated catalog, no vendor proxy
• Zero-infra - pip install proviz-elekto auto-starts the Rust server as a subprocess
• Any language - HTTP API, not a library binding
• Pluggable storage - SQLite (default) or PostgreSQL

Installation

ProvizElekto consists of a Rust server and various clients.

pip install proviz-elekto          # core only
pip install proviz-elekto[litellm] # + built-in LiteLLM integration

The proviz-server binary is bundled in the wheel.

CLI tool (proviz) is also included:

proviz --help

Documentation

• Catalog Setup — Seeding brands/models, adding rules, model groups
• Selection Algorithm — Scoring, headroom, priority, quality scores, retry hints
• HTTP API Reference — /select, /report, /complete, /health, /catalog/reload
• Deployment & Docker — Running the server, env vars, Docker, building from source
• Data Model — Table schemas for all pz_* tables

Quickstart

With LiteLLM (recommended)

from proviz_elekto import ProvizElekto

pz = ProvizElekto(db_path="./proviz.db")
# or PostgreSQL: pz = ProvizElekto(database_url=os.environ["DATABASE_URL"])

result = pz.call_litellm(
    step="verdict",
    messages=[{"role": "user", "content": "Summarize this document..."}],
    estimated_tokens=2500,
    requires_json_mode=True,
)
print(result.provider, result.candidate.model_slug, result.total_tokens)
# → mistral mistral-small-latest 312

call_litellm() selects the best available model, calls it, reports the outcome, and retries with the next eligible model on any failure — automatically.

Without litellm (server-side /complete)

The server calls the provider for you — no litellm or provider SDK in your environment. Best for thin/non-Python callers and minimal dependency footprints.

result = pz.complete(
    step="verdict",
    messages=[{"role": "user", "content": "Summarize this document..."}],
    estimated_tokens=2500,
    response_format={"type": "json_object"},
)
print(result.brand, result.model, result.prompt_tokens, result.completion_tokens, result.cost_usd)
# → mistral mistral-small-latest 2487 312 0.00031

complete() does select + provider call + report in a single round-trip. On provider failure it excludes the model and retries the next-best candidate server-side (up to 4 attempts). Pass tools=/tool_choice= to get un-executed tool_calls back and drive the tool loop yourself. Any OpenAI-compatible provider (groq, mistral, ovh, scaleway, novita, infomaniak, …) works.

Infomaniak embeds an account-specific product_id in its API URL. providers/infomaniak/brand.json stores it as a ${INFOMANIAK_PRODUCT_ID} placeholder in base_url, expanded from the environment at request time — set INFOMANIAK_PRODUCT_ID (from GET https://api.infomaniak.com/1/ai) next to INFOMANIAK_API_KEY wherever the server/Docker container runs. Any base_url in a provider's brand.json/models.json may use ${VAR} this way.

Non-USD pricing. A brand.json may set "price_currency" (e.g. "EUR" for Infomaniak); the model prices in its models.json are then in that currency. The selector converts everything to USD for cost scoring and for the cost_usd / actual_cost_usd figures using live ECB rates (from frankfurter.dev, fetched at most hourly, persisted so last-good values survive restarts). GET /fx/rates shows the current table. USD-only setups are unaffected.

The legacy /select + client-side call + /report flow (below) stays fully supported — use it when you want to own the provider call (streaming, custom SDK).

With a custom LLM caller

import anthropic

client = anthropic.Anthropic()

def my_llm(candidate):
    return client.messages.create(
        model=candidate.model_slug,
        max_tokens=1024,
        messages=[{"role": "user", "content": "Hello"}],
    )

result = pz.call("verdict", my_llm, estimated_tokens=100)
print(result.candidate.brand_slug, result.prompt_tokens)

Pass any callable that accepts a ModelCandidate and returns a response. ProvizElekto wraps it with the same select → report → retry loop.

Low-level API

If you need direct control over selection and reporting:

candidate = pz.select(step="verdict", estimated_tokens=2500)
try:
    response = my_llm_call(candidate)

    # Read provider rate-limit headers (Mistral/OpenAI style; Anthropic style also supported)
    hdrs = getattr(response, "_hidden_params", {}).get("additional_headers") or {}
    rem_req = hdrs.get("x-ratelimit-remaining-requests")
    rem_tok = hdrs.get("x-ratelimit-remaining-tokens")

    pz.report_success(
        candidate.model_id,
        estimated_tokens=candidate.estimated_tokens,  # releases in-flight reservation
        actual_tokens=response.usage.total_tokens,    # improves TPM window accuracy
        remaining_requests=int(rem_req) if rem_req is not None else None,
        remaining_tokens=int(rem_tok)   if rem_tok is not None else None,
    )
    # report_success is fire-and-forget — returns immediately, HTTP call runs in background
except RateLimitError as exc:
    msg = str(exc).lower()
    if "day" in msg or "daily" in msg:
        error_type = "tpd"
    elif "token" in msg:
        error_type = "tpm"
    else:
        error_type = "rpm"
    pz.report_rate_limit(candidate.model_id, error_type)  # synchronous — must complete before retry
except Exception:
    pz.report_error(candidate.model_id, "other")

estimated_tokens in each report call releases the in-flight reservation made at selection time. Omitting it is safe (legacy clients work unchanged) but leaves the in-flight counter inflated until the next selection clears it.

report_success is non-blocking: the HTTP call to proviz runs in a background daemon thread so the caller receives the LLM result without waiting for the round-trip. report_rate_limit and report_error remain synchronous because the model must be blocked in proviz before the retry select() call.

License

Apache-2.0