llm-market 0.2.0

Open clearinghouse for LLM bids, councils, and evidence-backed routing.

LLM Market

LLM Market is an open router for LLM work. Give it a prompt, model cards, prices, and optional prior evidence; it chooses the process with the best expected value: one model, a bid-informed single model, or a small council judged into one answer.

The point is simple: stop sending every prompt to the most expensive model when a cheaper route can match the answer.

Current Result

The latest local Codex eval in this repo uses six complex multi-step tasks: code review, source-grounded research, documentation synthesis, patch-policy review, eval diagnosis, and a server-gateway decision memo.

The first hard run was useful because it failed the product bar: LLM Market matched quality, but over-spent versus the best cheap single. After ingesting that run as evidence, the router learned to send the same task family directly to codex:gpt-5.4-mini:low.

Post-ingest result:

Run	Correct	Mean rubric score	Estimated cost
gpt-5.5:xhigh for every task	6 / 6	1.000	0.17546
gpt-5.4-mini:low for every task	6 / 6	0.976	0.0043865
LLM Market routed setup	6 / 6	0.979	0.00438725

That is equal quality at about 2.5% of the estimated xhigh cost on this complex-task slice. Versus the best cheap single, LLM Market is effectively the same cost after routing to that model directly.

Use this as a local evidence-loop result, not a broad benchmark claim. The public summary is tracked at outputs/current/codex_cost_smoke_summary.json; it records the aggregate result, limitations, and tracked model-card config, but not raw provider JSON, stderr, or full answer dumps. Those are left out because they can contain machine-specific details. Codex CLI does not return billing data, so production users should replace these estimates with their own provider prices.

Quickstart

python -m pip install llm-market
llm-market plan "Should this prompt use one model, bids, or a council?"

For local development:

python -m pip install -e ".[dev]"
python -m pytest

Inspect a route without calling a model:

llm-market plan "Should this prompt use one model, bids, or a council?"

Run locally with deterministic mock models:

llm-market run --mock --trace "Compare carbon taxes and cap-and-trade."

Mock answers are synthetic. Use them to inspect route mechanics, traces, and cost accounting before spending live provider calls.

The public package and CLI are llm-market. The Python import/module is llm_market because Python module names cannot contain hyphens, so python -m llm_market works too.

Run With Real Models

Codex CLI

Codex model ids use codex:<model-slug>:<reasoning-effort>, for example codex:gpt-5.5:low or codex:gpt-5.5:xhigh. The adapter shells out to an authenticated local codex exec.

Plan with the estimated-cost Codex cards:

llm-market plan \
  --config examples/codex_estimated_cost_models.json \
  "Compute 19 * 23 and give only the answer."

Run a live Codex call:

llm-market run \
  --provider codex \
  --config examples/codex_estimated_cost_models.json \
  --max-tokens 140 \
  "Compute 19 * 23 and give only the answer."

By default, Codex provider runs ignore user-level Codex config so evals stay reproducible. Pass --codex-use-user-config when you want the user's local Codex settings involved. Pass --codex-oss when you intentionally want Codex OSS or local-provider mode, such as Ollama or LM Studio.

Cursor Agent CLI

Cursor model ids use cursor:<cursor-model-slug>, for example cursor:gemini-3.1-pro or cursor:claude-4.6-opus-high. The adapter shells out to an authenticated local cursor agent --print run.

Check Cursor readiness:

cursor agent status
cursor agent models

Run live through Cursor's own model surface:

llm-market run \
  --provider cursor \
  --config examples/cursor_models.json \
  --max-tokens 400 \
  --trace \
  "Review this patch plan and say what is missing."

If you omit --config, --provider cursor uses a built-in starter Cursor model universe with Composer, Gemini, and Claude/Opus cards, not Cursor GPT/OpenAI cards. The starter costs are relative routing weights because Cursor does not expose stable public per-token prices through this CLI path. Replace them with your own observed costs, latency, and quality data when you have it. The starter Cursor cards include request_overhead_tokens: 60000, because Cursor Agent runs can report a large hidden prompt/workspace overhead even for short user prompts. Replace this reserve with your own observed overhead for your repo and Cursor workspace.

Cursor runs default to --cursor-mode ask and pass --trust so unattended skill runs do not stop at workspace-trust prompts. Use --cursor-mode plan for Cursor's planning mode, --cursor-no-trust when you want Cursor to enforce its trust prompt, and --cursor-sandbox enabled|disabled when you want to pass Cursor Agent's sandbox flag explicitly.

Mixed Local Agents

Use --provider multi when one run should mix local providers. The multiplexer dispatches by model id:

• codex:* goes to Codex CLI
• cursor:* goes to Cursor Agent CLI
• other OpenRouter-style ids go to OpenRouter

With no config, --provider multi uses the built-in Codex + Cursor starter cards:

llm-market run \
  --provider multi \
  --force-council \
  --max-tokens 500 \
  --trace \
  "Have a small council review this architecture choice."

Pass --config when you want a custom mixed universe or want to add OpenRouter models. OpenRouter calls still need OPENROUTER_API_KEY, but the multiplexer does not require that key unless an OpenRouter-style model is actually called.

OpenRouter

Set OPENROUTER_API_KEY, then use an OpenRouter model-card config:

export OPENROUTER_API_KEY=...

llm-market run \
  --provider openrouter \
  --config examples/openrouter_models.json \
  --trace \
  "Find the best route for this code-review prompt."

Codex Skill

This repo includes a Codex skill in skills/llm-market. Install that folder as ~/.codex/skills/llm-market to call LLM Market from any Codex thread:

python tools/install_integrations.py codex
python tools/install_integrations.py codex --check

The installer defaults to ${CODEX_HOME:-~/.codex}/skills/llm-market, deletes stale installed files that no longer exist in the repo, preserves local INSTALL_RECEIPT.txt, verifies the installed copy, and runs a no-spend wrapper smoke. Restart Codex after a sync so changed skill metadata is picked up.

The skill wrapper resolves LLM Market in this order:

1. an explicit --repo path
2. LLM_MARKET_REPO
3. the current repo or a parent directory
4. the skill directory or a parent directory
5. an installed Python package for plain CLI calls
6. a cached GitHub checkout when repo assets are requested or --update-github is used

If no checkout is found, the wrapper can clone the GitHub repo into $CODEX_HOME/llm-market/repo and run from that cached copy:

python ~/.codex/skills/llm-market/scripts/llm_market_cli.py \
  --update-github \
  plan \
  "Should this prompt use one model, bids, or a council?"

Inside Codex, plain skill calls default to a skill-local Codex-only model-card file, so they do not select OpenRouter or other non-Codex model ids unless you explicitly pass --config or an explicit non-Codex provider. Plain skill live-eval calls also default to --provider codex. Use --config when you intentionally want a custom, OpenRouter, or server-side provider universe.

Run the read-only provider doctor before Codex local-provider work:

python ~/.codex/skills/llm-market/scripts/codex_provider_doctor.py

The skill can distribute the package, examples, eval packs, and wrapper from GitHub. Live calls still need local provider state: Codex installed and authenticated, Cursor CLI installed and authenticated, or OpenRouter/OpenAI- compatible API keys. The skill reports local provider readiness; it does not edit ~/.codex/config.toml.

Check all local integration surfaces from the repo:

python tools/install_integrations.py all --check

Today this means a Codex skill sync/check, a no-spend Cursor provider plan smoke using the built-in Cursor starter cards, and an explicit Claude Code pending notice. Claude models are available through OpenRouter-style cards and Cursor cards; a Claude Code local-provider adapter is not checked in yet.

PyPI is not required for the Codex skill. The wrapper can clone the GitHub repo and run LLM Market from source. The PyPI package is useful for plain CLI calls and ordinary Python installs with pip install llm-market.

How Routing Works

LLM Market makes process selection explicit:

1. QueryAnalyzer profiles the prompt for task dimensions, complexity, risk, recency, context size, and council tendency.
2. MarketRouter scores model cards by capability fit, prior evidence, reliability, cost, latency, context window, and diversity.
3. With --market auto, non-routine prompts collect compact bids before the final route is locked.
4. The router chooses a single model when one route has earned it by utility, evidence, or bid dominance.
5. The router chooses a council when the evidence is mixed, bids are close, or the prompt needs synthesis, critique, or coverage.
6. MarketEngine executes the route, applies fallbacks, and records a trace.
7. MarketJudge merges council answers, checks accepted bid promises, preserves useful unique findings, and returns the final answer.

Prompt profiling is a weak prior. Bids, evidence, budgets, and observed outcomes are the load-bearing signals.

Configuration

Model cards are editable beliefs about cost, latency, reliability, context, and capability:

{
  "models": [
    {
      "id": "openai/gpt-latest",
      "provider": "openai",
      "capabilities": {
        "coding": 0.94,
        "reasoning": 0.92,
        "writing": 0.82
      },
      "input_cost_per_mtok": 5.0,
      "output_cost_per_mtok": 15.0,
      "context_tokens": 256000,
      "latency_ms": 2500,
      "reliability": 0.94,
      "tags": ["structured", "tool-use", "generalist"]
    }
  ]
}

Evidence rows record what worked on prior prompts:

{"model":"openai/gpt-4.1-mini","dimensions":{"coding":1.0},"score":1.0,"correct":true,"complexity":0.32,"risk":0.12}

Exact or strongly similar prompt history counts more than broad task-family averages. A single model needs enough margin before it can suppress council routing.

Python API

import asyncio

from llm_market import HeuristicMarketJudge, MarketEngine, MockModelClient
from llm_market.presets import default_model_cards

client = MockModelClient.from_static(
    {
        "openai/gpt-latest": "Strong coding and structured reasoning answer.",
        "anthropic/claude-opus-latest": "Careful critique and risk analysis.",
        "google/gemini-pro-latest": "Broad research and long-context synthesis.",
    }
)

engine = MarketEngine(
    model_cards=default_model_cards(),
    client=client,
    judge=HeuristicMarketJudge(),
)

result = asyncio.run(engine.run("Compare two approaches to a risky migration."))
print(result.answer)
print(result.trace.route.to_dict())

CLI Reference

plan explains the route without model calls:

llm-market plan --format json \
  --expected-output-tokens 800 \
  --max-expected-cost-usd 0.02 \
  "Write a SQL query for weekly retention."

llm-market plan --provider cursor \
  "Should Cursor handle this prompt with one model or a council?"

llm-market plan --provider multi \
  "Should local Codex and Cursor agents review this architecture together?"

Installed-package fallback is intentionally narrower: it can run the CLI and default in-package model cards, but repo assets such as examples/, outputs/, and skills/ require a source checkout or the cached GitHub checkout. The real-model commands below assume you are in a source checkout or the skill has resolved a cached GitHub checkout.

run executes a route:

llm-market run --mock --trace "Should this question use a council?"
llm-market run --mock --market auto --trace "Compare these migration options."
llm-market run --mock --council off --trace "Force single-model routing."

Presets set the operating posture without changing model cards:

• --preset fast: single-model, no bids, no council; uses evidence and cost pressure to prefer cheap earned routes.
• --preset balanced: default evidence/cost-aware routing.
• --preset max: bid-first and more quality-seeking; allows larger councils, one fallback, and lower cost/latency penalties without blindly forcing a council.

llm-market run \
  --provider codex \
  --config examples/codex_estimated_cost_models.json \
  --evidence outputs/current/evidence_codex_current.jsonl \
  --preset fast \
  "Summarize this diff and flag likely bugs."

live-eval calls real providers and compares the routed answer with baselines:

llm-market live-eval \
  --provider codex \
  --config examples/codex_estimated_cost_models.json \
  --evidence outputs/current/evidence_codex_current.jsonl \
  --tasks examples/codex_complex_multistep_tasks.jsonl \
  --baseline-mode external \
  --baseline-config examples/codex_complex_baselines.json \
  --provider-timeout-seconds 240 \
  --max-tokens 700 \
  --progress \
  --output outputs/archive/new_codex_run.json

Use --baseline-mode external --baseline-config ... when you want a clean comparison against direct baselines such as xhigh-for-everything and cheap fixed-single arms without direct-running every selectable candidate. Omit --baseline-mode to keep the default full-baseline behavior.

For objective eval tasks with graders, live-eval also has an experimental validation ladder. It is off by default. When enabled, the routed answer is graded first; only a failed grade can trigger same-route retry or forced-council escalation:

llm-market live-eval \
  --provider codex \
  --config examples/codex_estimated_cost_models.json \
  --tasks examples/codex_complex_multistep_tasks.jsonl \
  --baseline-mode none \
  --validation-ladder same-route-then-council

Use --best-single auto for product routing. When similar prior outcomes show that a selectable model repeatedly dominates, the router can use that model directly.

evidence-ingest turns live-eval artifacts into future routing evidence:

llm-market evidence-ingest \
  --results outputs/archive/<new-run>.json \
  --output outputs/current/evidence_codex_current.jsonl

Evidence can include prompt text, run identity, latency, route metadata, and model outcomes. Redact or keep private any eval artifact that contains customer, repo, or personal task text before publishing an evidence store.

route-episodes export preserves richer scored routing decisions for local inspection, shadow evaluation, and future calibration or training:

llm-market live-eval \
  --provider codex \
  --config examples/codex_estimated_cost_models.json \
  --tasks examples/codex_complex_multistep_tasks.jsonl \
  --baseline-mode selectable \
  --output ~/.llm-market/runs/<run-id>.json

llm-market route-episodes export \
  --results ~/.llm-market/runs/<run-id>.json \
  --output ~/.llm-market/route_episodes.jsonl

Compact evidence JSONL is the runtime prior for today's router. Route episodes JSONL is a richer private artifact with prompt profile, candidates, policy, route action, scored outcome, baselines, and counterfactuals. Route episodes are local/private by default: LLM Market does not upload, sync, phone home, collect telemetry, or share them automatically. Use --redact-prompt when exporting a dedupable prompt-hash-only copy. Future global route models should be trained only from maintainer-controlled, public/shareable, or explicit opt-in artifacts.

Installed packages include two portable Codex route-world advisors. The legacy builtin:codex-route-world-v1 loads the bundled sanitized JEPA sufficiency runtime internally, so testers do not need a zip, VEI checkout, or local training folder. The bundled sufficiency readout uses generic profile, candidate, and JEPA summary features; prompt-keyword feature buckets are excluded from the packaged model. This installed v1 artifact declares input_contract=profile_augmented; it is a portable bridge, not the final zero-hand-shaped JEPA-only route policy.

The stricter promoted contract is jepa_state_action: the task state comes from learned JEPA/world features, while the only non-world inputs are measured route-action descriptors such as candidate cost/count or bid availability. Deterministic prompt/task analyzer fields such as complexity, risk, council_score, and dim_* are rejected by that contract.

builtin:codex-route-world-v2 is the first stricter jepa_state_action promotion. It is deliberately narrower than full routing: a Gate 2 accept/continue scorer that runs only after the first cheap or primary answer exists. The packaged v2 scorer is a residual readout: it starts from the per-action Gate 2 baseline, then asks the JEPA model whether this prompt and draft should deviate from that average. It uses learned prompt/draft embeddings, JEPA predicted latents, route-action descriptors, and sanitized draft state; it logs hashes and lengths, not raw drafts.

The bundled v2 artifact is the seed72 residual Gate 2 model. On the fresh Packet 13 challenge holdout it matched always-continue success while continuing 61 / 120 decisions instead of 120 / 120: 82 successes, 0 missed rescues, 1 false-continue regression, and $0.18370975 estimated extra continuation cost. Full route-program selection, council planning, and market-maker bidding remain maintainer-local shadow experiments until they have their own fresh-heldout evidence. v2 does not need VEI at runtime, but actual JEPA inference needs the optional runtime extra:

python3 -m pip install --upgrade "llm-market[jepa]>=0.2.0"

llm-market route-world list-bundled
llm-market route-sufficiency list-bundled

llm-market plan \
  --provider codex \
  --world-model builtin:codex-route-world-v1 \
  --world-routing-mode shadow \
  --format json \
  "Review this repo change and pick the cheapest safe route."

For v2, use a route that creates a first answer plus a possible continuation. In shadow, it records the Gate 2 accept/continue recommendation; in apply, it may stop after the first answer or let the retry/review continuation run:

llm-market live-eval \
  --provider codex \
  --config examples/codex_estimated_cost_models.json \
  --world-model builtin:codex-route-world-v2 \
  --world-routing-mode shadow \
  --tasks examples/codex_complex_multistep_tasks.jsonl \
  --baseline-mode none \
  --validation-ladder same-route

shadow records route_world diagnostics without changing the route. Use --world-routing-mode apply only when you intentionally want the advisor to change route decisions. Apply-mode advice is still constrained by evidence: the advisor can only apply a single-model recommendation when prior observations identify that same model as the best observed single. Route-world advice can also suppress or request a market bid round when the candidate runtime supports that signal and the user has not explicitly disabled the market. During maintainer experiments, compiled policy-readout and market-maker readout candidates can be tested from ignored local paths before any package promotion. Market-maker readouts can optionally wrap a bundled sufficiency model and use its learned world_* scores as pre-bid features without adding VEI or training dependencies to the install path. Local candidates can also be tested with explicit apply; single-model downgrades still require evidence support, while council-veto and market-open advice can be applied as guarded runtime signals. Local gate2_readout_experimental.v1 candidates can be tested the same way: shadow logs an accept/continue recommendation, while apply can stop after the first answer or let the retry/review continuation run.

llm-market live-eval \
  --provider codex \
  --config examples/codex_estimated_cost_models.json \
  --world-model path/to/ignored-lab/compiled/candidate-name/route_world_model.json \
  --world-routing-mode shadow \
  --tasks examples/codex_complex_multistep_tasks.jsonl \
  --baseline-mode none \
  --heuristic-judge

Advanced route-policy experiments can also override --market, --council-threshold, --single-utility-margin, and --cost-sensitivity from the CLI before any candidate is promoted.

Training runs, raw route rows, prompts, JEPA exports, and experiment matrices belong in ignored local lab folders. The repo should only receive the promoted portable runtime artifact and the package code that loads it. For packaged artifact details, tester commands, and the future model refresh process, see docs/bundled_jepa_model.md.

Repo Map

• src/llm_market/analysis.py: prompt profiling.
• src/llm_market/routing.py: utility, cost, latency, diversity, and judge selection.
• src/llm_market/bidding.py: model self-bids for market routing.
• src/llm_market/evidence.py: prior-result observations and evidence deltas.
• src/llm_market/route_episodes.py: richer local route episode export schema.
• src/llm_market/route_world.py: bundled/local route-world advisor runtime.
• src/llm_market/sufficiency.py: JEPA-gated cheap-route sufficiency advisor.
• src/llm_market/engine.py: route execution and fallback behavior.
• src/llm_market/judge.py: heuristic judge and LLM judge.
• src/llm_market/clients.py: OpenRouter/OpenAI-compatible client and Codex CLI client.
• examples/codex_estimated_cost_models.json: cost-aware Codex model cards.
• examples/codex_complex_multistep_tasks.jsonl: harder rubric-graded Codex eval tasks.
• examples/codex_complex_baselines.json: direct xhigh and cheap-single baselines for complex-task evals.
• outputs/current/codex_cost_smoke_summary.json: tracked summary for the README result.
• outputs/current: current evidence packet and active evidence JSONL.
• outputs/archive: archived run-specific reports behind the current packet.
• skills/llm-market: Codex skill wrapper and integration notes.
• tools/install_integrations.py: sync/check local integration surfaces.

OpenRouter Fusion Relationship

OpenRouter Fusion made the council primitive legible: ask multiple models, have a judge compare answers, then produce one response. LLM Market keeps that primitive open and adds the route policy around it: when to pay for a council, when to ask for bids, when a cheap model is enough, when prior evidence should promote one model, and what trace data should be saved for the next prompt.

Scope

• Codex cost comparisons use estimated relative costs because Codex CLI does not expose per-call billing.
• Provider-specific cache pricing, KV-cache accounting, and cache-hit discounts are future cost-model work.
• The included live result is a small complex-task eval. Run your own task families before trusting any routing policy in production.
• Model cards need current prices, reliability estimates, and observed evidence to stay useful.

Third-Party Notices

See THIRD_PARTY_NOTICES.md.