open-ultra

A self-training LLM routing proxy for agentic CLIs.

Goal: match your frontier model's output quality at a fraction of the cost. Run your coding-agent traffic through open-ultra. A hidden shadow lane re-runs your requests on cheap open models, an agentic grader scores them against what your frontier model actually did, and once a cheap model has proven it can match the frontier on a kind of task, the router starts sending that traffic to the cheap model instead. You keep the same quality on the work cheap models handle well, and only pay frontier prices for the work that actually needs it. No benchmarks, no synthetic data: it calibrates to your live work.

Routing stays off until a trained router passes evaluation, and a cheap model only earns traffic where it scored at least as well as the frontier. Until then open-ultra is a transparent passthrough that quietly collects evidence.

How it works

your agent ──► proxy (byte passthrough to your frontier model, zero added latency)
                 │
                 ├─► store: primary request/response pairs
                 │
                 └─► shadow lane (always on while serving, invisible to you)
                       ├─ response-only mode: identical request bytes → shadow model(s)
                       └─ worktree-exec mode: shadow model works agentically in a
                          throwaway git worktree, tools confined there, diff captured
                             │
                       agentic grader: scores shadow vs primary (answer vs answer,
                       or worktree diff vs your real diff) → pairs.jsonl
                             │
                train ──► router (per-model capability heads) ──► route on

• Primary lane: your request is forwarded byte-for-byte, auth headers included. Your agent behaves exactly as before.
• Shadow lane: the same request goes to the cheap models in your pool, off the hot path. In worktree-exec mode the shadow model gets a real (isolated) copy of your repo and an agent loop, so agentic ability is graded, not just prose.
• Grader: a configurable judge model compares shadow output against the frontier output that actually served you. Same context, same granularity as real routing decisions.
• Router: one shared classifier with an independent capability head per pool model. Eligible = heads above threshold; the proxy picks the cheapest eligible model that is currently up, otherwise the frontier. Prices and availability are config, never weights.

Supported agents (day 1)

The proxy speaks Anthropic Messages and OpenAI (Chat Completions + Responses) inbound. Outbound is OpenAI-compatible only, which covers OpenRouter, vLLM, Ollama, SGLang, Together, DeepSeek, and effectively every serving stack. Shadow models, the grader, and the frontier are each just an endpoint entry in pool.json.

Codex retries aggressively and can trip per-minute rate limits on :free OpenRouter models. Point its shadow/routing models at a paid or self-hosted endpoint for smooth runs.

Install

open-ultra is a uv tool. Install the CLI (it lands on your PATH as openultra):

# from PyPI (once published)
uv tool install open-ultra

# or straight from the repo
uv tool install "git+https://github.com/numinous-technology/open-ultra"

# or from a local checkout (editable, for development)
git clone https://github.com/numinous-technology/open-ultra
cd open-ultra
uv tool install --editable .

Upgrade with uv tool upgrade open-ultra, remove with uv tool uninstall open-ultra.

Quickstart

openultra init                # one-time: API key, pick a pool preset, connectivity check
openultra claude              # auto-starts the daemon, launches Claude Code attached
# ... work normally for a while ...
openultra report              # per-model as-good rates, projected savings
openultra train               # fit the router on accumulated pairs
openultra route on            # start routing eligible traffic to cheap models

openultra init writes ~/.openultra/{env,config.json,pool.json}. Presets: free (rate-limited $0 models to try it out), budget (DeepSeek V4 Flash), balanced (Flash + MiniMax M3).

Attaching your agent

openultra up runs the daemon, but nothing attaches automatically: a CLI is attached only when its base URL points at the proxy. Three levels:

1. Per session: openultra claude / openultra codex / openultra opencode. Thin wrappers that start the daemon if needed and launch the CLI with the base URL injected for that process only. Plain claude still bypasses.
2. Persistent: openultra attach claude writes ANTHROPIC_BASE_URL into ~/.claude/settings.json (similarly Codex config.toml, OpenCode config). Every future run attaches. openultra detach claude undoes it.
3. Manual / any app: run openultra up and point anything at http://localhost:8484. The daemon speaks /v1/messages (Anthropic) and /v1/chat/completions + /v1/responses (OpenAI), so any SDK with a base_url override works, not just agent CLIs.

CLI

Every command and subcommand supports -h/--help.

Data model

~/.openultra/
  pool.json          # frontier + shadow models: endpoint, price, enabled
  traffic.jsonl      # primary lane request/response log
  pairs.jsonl        # {request_hash, shadow_model, mode, grade, margin, costs}
  router/            # trained weights, thresholds, eval report

Design invariants

1. Byte-exact training data: the router trains on the literal requests it will route. No reconstruction, no synthetic prompts.
2. Default to frontier: cheap models earn traffic through graded evidence on your workload. Worst case is the status quo.
3. Pool is config, never code: prices, endpoints, and availability live in pool.json. Adding a model means shadowing it for a while, then retraining.
4. Invisible shadow: the shadow lane never touches your session, your files (worktrees are throwaway and isolated, network off by default), or your latency.
5. Local first: traffic goes only to endpoints you configured. Logs stay on your machine.

Cost

The shadow lane roughly doubles request volume, but on models that cost 1-3% of frontier prices. Worktree-exec mode adds local CPU and disk. A spend cap in pool.json bounds the shadow budget; the lane pauses when the cap is hit.

Repo layout

src/open_ultra/
  cli.py                 # all verbs
  init_wizard.py         # openultra init
  paths.py               # ~/.openultra locations (OPENULTRA_HOME override)
  proxy/
    server.py            # inbound Anthropic + OpenAI dialects, passthrough + capture + routing
    route_exec.py        # serve a routed request from a cheap model in the caller's dialect
    parse.py             # normalize captured responses (plain + SSE, all dialects)
  shadow/runner.py       # replay captured requests on shadow models
  grader/judge.py        # blinded A/B grading -> pairs.jsonl
  clients/translate.py   # Anthropic / Responses -> Chat Completions
  render.py              # shared request rendering (grader + router see the same text)
  router/
    train.py             # per-model capability heads, conversation-grouped split, eval gate
    infer.py             # versioned router loading + routing decision
  eval/report.py         # receipts
  control.py             # daemon lifecycle + config edits
tests/e2e.py             # offline 23-check end-to-end suite

Development

uv sync
uv run python tests/e2e.py    # offline end-to-end test: mock upstream, isolated OPENULTRA_HOME
uv build                      # build wheel + sdist into dist/

Set OPENULTRA_HOME to relocate the data directory, OPENULTRA_URL if the daemon runs on a non-default address.

Publishing (maintainers): uv build && uv publish (needs a PyPI token in UV_PUBLISH_TOKEN).

Roadmap

Everything below the divider is built and tested; the whole pipeline runs end to end against real Claude Code, Codex, and OpenCode traffic. What remains is mostly depth: better routing features and grading modes.

Proxy & capture

• Byte-exact passthrough to the frontier, zero added latency
• Anthropic Messages inbound (Claude Code)
• OpenAI Chat Completions inbound
• OpenAI Responses inbound (Codex)
• Streaming (SSE) capture for every dialect, tee'd without buffering the client
• Request/response capture to traffic.jsonl (headers never persisted)
• Localhost-only bind, ~/.openultra/env at 0600

Shadow lane

• Always-on shadow queue drained by a background worker
• Anthropic → OpenAI-compatible request translation (system, tools, tool_use/tool_result)
• Responses → Chat Completions translation
• Per-model cost estimation, transient-error retries with attempt cap
• Per-lane enable flags (shadow vs route, independent)
• Worktree-exec grading mode (shadow model runs agentically in an isolated git worktree)
• Shadow spend cap that pauses the lane

Grader

• Blinded, order-shuffled A/B judging → pairs.jsonl
• Tool-call-aware scoring (compares proposed actions, not just prose)
• Configurable judge endpoint
• Delayed grading that uses next-turn ground truth (tests failed, user corrected)

Router

• Per-model capability heads (one classifier, independent sigmoid per model)
• Conversation-grouped time split (no near-duplicate leakage)
• Precision gate: a model is routable only if it clears eval precision
• Cheapest-eligible-and-available routing, cost order from config not weights
• Versioned routers (train --name, routers list/show/note/use/rm), hot-swap + rollback
• Live routing with transparent frontier fallback on any failure
• Stronger features / small-LM feature tier
• Epsilon exploration so cheap models keep earning re-verification once routing is on

CLI & UX

• init wizard (API key, pool preset, connectivity test)
• up / down / status, config hot-reload
• models add with OpenRouter price autofill; list/rm/enable/disable
• report receipts (as-good rate by turn type, projected savings)
• train / route / shadow / routers
• claude / codex / opencode launch wrappers + attach / detach
• --help on every command, --version
• uv tool install, wheel builds + installs clean in a fresh venv
• Offline end-to-end test suite (23 checks, mock upstream)
• Publish to PyPI
• Systemd/launchd service unit

The only thing between you and savings: run it on your traffic long enough to train a model that passes the gate. Everything to collect, grade, train, and route is done.

License

Apache-2.0