proviz-elekto 0.9.0

Smart LLM model router - auto-starts proviz-server binary, no Docker required

ProvizElekto

Smart LLM model router. Picks the best model for each call based on context size, rate limits, and capabilities — and retries automatically on failure.

Your app → pz.call(step, fn)               → CallResult
           pz.call_litellm(step, messages) → CallResult
                    ↕  (automatic)
           select → LLM call → report → retry on failure
                    ↕
              proviz-server (Rust)
          rate-limit state · catalog

Key difference from LiteLLM fallback: LiteLLM retries after failure. ProvizElekto picks the right model before the call — skipping models that are rate-limited or near their quota, can't fit the context, or lack required capabilities — then retries with the next eligible model automatically.

Two roles depending on the path

In the regular flow, the server is a pure router — it picks the model and returns credentials; your code makes the actual LLM call.

In the batch flow, the server becomes the caller:

# Regular: YOUR code calls the LLM
Your app → POST /select → ModelCandidate → your code → Mistral/OpenAI/...
                                                ↓
                                        POST /report

# Batch: the SERVER calls Mistral on your behalf
Worker A ──┐
Worker B ──┤ POST /batch/submit → server accumulates over window_secs
Worker C ──┘
                    ↓ server → POST Mistral /v1/batch/jobs (50% discount)
                    ↓ server polls until complete
Worker A ──┐
Worker B ──┤ GET /batch/result/{id} → response
Worker C ──┘

The batch path pools requests from all workers into a single Mistral job — the only way to qualify for Mistral's 50% batch discount. No individual worker can do this on its own, so the server acts as the aggregation point and makes the Mistral call itself.

Deployment note: when using batch, the server process (including Docker) must have the Mistral API key env vars set. In the regular flow, API keys only need to be present in the caller's environment.

Features

• Context-aware selection - don't waste a 128k model on a 1k prompt
• Proactive quota tracking - sliding-window counters (RPM/TPM/RPD/TPD) plus atomic in-flight reservations; avoids over-booking before any 429 fires
• Provider-anchored windows - every successful call forwards x-ratelimit-remaining-* headers back to the server; the window floor is clamped to provider reality so internal estimates can't drift below what the provider actually sees
• Scored selection - multi-component scoring: fast headroom (RPS/RPM/TPM, 25%), daily budget (RPD/TPD, 20%), quality (20%), cost (15%), latency (10%), traffic balance (10%). Over-quota models stay eligible with lower scores — AllModelsExhausted only fires when every model is in reactive 429 cooldown.
• Traffic shaping - per-brand traffic_weight steers load proportionally across providers in a 5-minute rolling window; under-served brands get a higher score on the traffic component
• Capability filtering - hard requirements for function calling, JSON mode
• Quality floor - reject models below a quality threshold per step
• Model groups - define named pools of models (e.g. "fast-chat", "coding-tier1") and restrict selection to that pool
• Your keys, your models - curated catalog, no vendor proxy
• Zero-infra - pip install proviz-elekto auto-starts the Rust server as a subprocess
• Any language - HTTP API, not a library binding
• Pluggable storage - SQLite (default) or PostgreSQL

Installation

ProvizElekto consists of a Rust server and various clients.

pip install proviz-elekto          # core only
pip install proviz-elekto[litellm] # + built-in LiteLLM integration

The proviz-server binary is bundled in the wheel.

CLI tool (proviz) is also included:

proviz --help

Documentation

• Catalog Setup — Seeding brands/models, adding rules, model groups
• Selection Algorithm — Scoring, headroom, priority, quality scores, retry hints
• HTTP API Reference — /select, /report, /health, /catalog/reload
• Deployment & Docker — Running the server, env vars, Docker, building from source
• Data Model — Table schemas for all pz_* tables

Quickstart

With LiteLLM (recommended)

from proviz_elekto import ProvizElekto

pz = ProvizElekto(db_path="./proviz.db")
# or PostgreSQL: pz = ProvizElekto(database_url=os.environ["DATABASE_URL"])

result = pz.call_litellm(
    step="verdict",
    messages=[{"role": "user", "content": "Summarize this document..."}],
    estimated_tokens=2500,
    requires_json_mode=True,
)
print(result.provider, result.candidate.model_slug, result.total_tokens)
# → mistral mistral-small-latest 312

call_litellm() selects the best available model, calls it, reports the outcome, and retries with the next eligible model on any failure — automatically.

With a custom LLM caller

import anthropic

client = anthropic.Anthropic()

def my_llm(candidate):
    return client.messages.create(
        model=candidate.model_slug,
        max_tokens=1024,
        messages=[{"role": "user", "content": "Hello"}],
    )

result = pz.call("verdict", my_llm, estimated_tokens=100)
print(result.candidate.brand_slug, result.prompt_tokens)

Pass any callable that accepts a ModelCandidate and returns a response. ProvizElekto wraps it with the same select → report → retry loop.

Low-level API

If you need direct control over selection and reporting:

candidate = pz.select(step="verdict", estimated_tokens=2500)
try:
    response = my_llm_call(candidate)

    # Read provider rate-limit headers (Mistral/OpenAI style; Anthropic style also supported)
    hdrs = getattr(response, "_hidden_params", {}).get("additional_headers") or {}
    rem_req = hdrs.get("x-ratelimit-remaining-requests")
    rem_tok = hdrs.get("x-ratelimit-remaining-tokens")

    pz.report_success(
        candidate.model_id,
        estimated_tokens=candidate.estimated_tokens,  # releases in-flight reservation
        actual_tokens=response.usage.total_tokens,    # improves TPM window accuracy
        remaining_requests=int(rem_req) if rem_req is not None else None,
        remaining_tokens=int(rem_tok)   if rem_tok is not None else None,
    )
    # report_success is fire-and-forget — returns immediately, HTTP call runs in background
except RateLimitError as exc:
    msg = str(exc).lower()
    if "day" in msg or "daily" in msg:
        error_type = "tpd"
    elif "token" in msg:
        error_type = "tpm"
    else:
        error_type = "rpm"
    pz.report_rate_limit(candidate.model_id, error_type)  # synchronous — must complete before retry
except Exception:
    pz.report_error(candidate.model_id, "other")

estimated_tokens in each report call releases the in-flight reservation made at selection time. Omitting it is safe (legacy clients work unchanged) but leaves the in-flight counter inflated until the next selection clears it.

report_success is non-blocking: the HTTP call to proviz runs in a background daemon thread so the caller receives the LLM result without waiting for the round-trip. report_rate_limit and report_error remain synchronous because the model must be blocked in proviz before the retry select() call.

License

Apache-2.0