llm-market 0.1.1

Open clearinghouse for LLM bids, councils, and evidence-backed routing.

LLM Market

LLM Market is an open router for LLM work. Give it a prompt, model cards, prices, and optional prior evidence; it chooses the process with the best expected value: one model, a bid-informed single model, or a small council judged into one answer.

The point is simple: stop sending every prompt to the most expensive model when a cheaper route can match the answer.

Current Result

The latest local Codex eval in this repo uses six complex multi-step tasks: code review, source-grounded research, documentation synthesis, patch-policy review, eval diagnosis, and a server-gateway decision memo.

The first hard run was useful because it failed the product bar: LLM Market matched quality, but over-spent versus the best cheap single. After ingesting that run as evidence, the router learned to send the same task family directly to codex:gpt-5.4-mini:low.

Post-ingest result:

Run	Correct	Mean rubric score	Estimated cost
gpt-5.5:xhigh for every task	6 / 6	1.000	0.17546
gpt-5.4-mini:low for every task	6 / 6	0.976	0.0043865
LLM Market routed setup	6 / 6	0.979	0.00438725

That is equal quality at about 2.5% of the estimated xhigh cost on this complex-task slice. Versus the best cheap single, LLM Market is effectively the same cost after routing to that model directly.

Use this as a local evidence-loop result, not a broad benchmark claim. The public summary is tracked at outputs/current/codex_cost_smoke_summary.json; it records the aggregate result, limitations, and tracked model-card config, but not raw provider JSON, stderr, or full answer dumps. Those are left out because they can contain machine-specific details. Codex CLI does not return billing data, so production users should replace these estimates with their own provider prices.

Quickstart

python -m pip install llm-market
llm-market plan "Should this prompt use one model, bids, or a council?"

For local development:

python -m pip install -e ".[dev]"
python -m pytest

Inspect a route without calling a model:

llm-market plan "Should this prompt use one model, bids, or a council?"

Run locally with deterministic mock models:

llm-market run --mock --trace "Compare carbon taxes and cap-and-trade."

Mock answers are synthetic. Use them to inspect route mechanics, traces, and cost accounting before spending live provider calls.

The public package and CLI are llm-market. The Python import/module is llm_market because Python module names cannot contain hyphens, so python -m llm_market works too.

Run With Real Models

Codex CLI

Codex model ids use codex:<model-slug>:<reasoning-effort>, for example codex:gpt-5.5:low or codex:gpt-5.5:xhigh. The adapter shells out to an authenticated local codex exec.

Plan with the estimated-cost Codex cards:

llm-market plan \
  --config examples/codex_estimated_cost_models.json \
  "Compute 19 * 23 and give only the answer."

Run a live Codex call:

llm-market run \
  --provider codex \
  --config examples/codex_estimated_cost_models.json \
  --max-tokens 140 \
  "Compute 19 * 23 and give only the answer."

By default, Codex provider runs ignore user-level Codex config so evals stay reproducible. Pass --codex-use-user-config when you want the user's local Codex settings involved. Pass --codex-oss when you intentionally want Codex OSS or local-provider mode, such as Ollama or LM Studio.

OpenRouter

Set OPENROUTER_API_KEY, then use an OpenRouter model-card config:

export OPENROUTER_API_KEY=...

llm-market run \
  --provider openrouter \
  --config examples/openrouter_models.json \
  --trace \
  "Find the best route for this code-review prompt."

Codex Skill

This repo includes a Codex skill in skills/llm-market. Install that folder as ~/.codex/skills/llm-market to call LLM Market from any Codex thread:

mkdir -p ~/.codex/skills/llm-market
cp -R skills/llm-market/. ~/.codex/skills/llm-market/

The skill wrapper resolves LLM Market in this order:

1. an explicit --repo path
2. LLM_MARKET_REPO
3. the current repo or a parent directory
4. the skill directory or a parent directory
5. an installed Python package for plain CLI calls
6. a cached GitHub checkout when repo assets are requested or --update-github is used

If no checkout is found, the wrapper can clone the GitHub repo into $CODEX_HOME/llm-market/repo and run from that cached copy:

python ~/.codex/skills/llm-market/scripts/llm_market_cli.py \
  --update-github \
  plan \
  --config examples/codex_estimated_cost_models.json \
  "Should this prompt use one model, bids, or a council?"

Run the read-only provider doctor before Codex local-provider work:

python ~/.codex/skills/llm-market/scripts/codex_provider_doctor.py

The skill can distribute the package, examples, eval packs, and wrapper from GitHub. Live calls still need local provider state: Codex installed and authenticated, or OpenRouter/OpenAI-compatible API keys. The skill reports local provider readiness; it does not edit ~/.codex/config.toml.

PyPI is not required for the Codex skill. The wrapper can clone the GitHub repo and run LLM Market from source. The PyPI package is useful for plain CLI calls and ordinary Python installs with pip install llm-market.

How Routing Works

LLM Market makes process selection explicit:

1. QueryAnalyzer profiles the prompt for task dimensions, complexity, risk, recency, context size, and council tendency.
2. MarketRouter scores model cards by capability fit, prior evidence, reliability, cost, latency, context window, and diversity.
3. With --market auto, non-routine prompts collect compact bids before the final route is locked.
4. The router chooses a single model when one route has earned it by utility, evidence, or bid dominance.
5. The router chooses a council when the evidence is mixed, bids are close, or the prompt needs synthesis, critique, or coverage.
6. MarketEngine executes the route, applies fallbacks, and records a trace.
7. MarketJudge merges council answers, checks accepted bid promises, preserves useful unique findings, and returns the final answer.

Prompt profiling is a weak prior. Bids, evidence, budgets, and observed outcomes are the load-bearing signals.

Configuration

Model cards are editable beliefs about cost, latency, reliability, context, and capability:

{
  "models": [
    {
      "id": "openai/gpt-latest",
      "provider": "openai",
      "capabilities": {
        "coding": 0.94,
        "reasoning": 0.92,
        "writing": 0.82
      },
      "input_cost_per_mtok": 5.0,
      "output_cost_per_mtok": 15.0,
      "context_tokens": 256000,
      "latency_ms": 2500,
      "reliability": 0.94,
      "tags": ["structured", "tool-use", "generalist"]
    }
  ]
}

Evidence rows record what worked on prior prompts:

{"model":"openai/gpt-4.1-mini","dimensions":{"coding":1.0},"score":1.0,"correct":true,"complexity":0.32,"risk":0.12}

Exact or strongly similar prompt history counts more than broad task-family averages. A single model needs enough margin before it can suppress council routing.

Python API

import asyncio

from llm_market import HeuristicMarketJudge, MarketEngine, MockModelClient
from llm_market.presets import default_model_cards

client = MockModelClient.from_static(
    {
        "openai/gpt-latest": "Strong coding and structured reasoning answer.",
        "anthropic/claude-opus-latest": "Careful critique and risk analysis.",
        "google/gemini-pro-latest": "Broad research and long-context synthesis.",
    }
)

engine = MarketEngine(
    model_cards=default_model_cards(),
    client=client,
    judge=HeuristicMarketJudge(),
)

result = asyncio.run(engine.run("Compare two approaches to a risky migration."))
print(result.answer)
print(result.trace.route.to_dict())

CLI Reference

plan explains the route without model calls:

llm-market plan --format json \
  --expected-output-tokens 800 \
  --max-expected-cost-usd 0.02 \
  "Write a SQL query for weekly retention."

Installed-package fallback is intentionally narrower: it can run the CLI and default in-package model cards, but repo assets such as examples/, outputs/, and skills/ require a source checkout or the cached GitHub checkout. The real-model commands below assume you are in a source checkout or the skill has resolved a cached GitHub checkout.

run executes a route:

llm-market run --mock --trace "Should this question use a council?"
llm-market run --mock --market auto --trace "Compare these migration options."
llm-market run --mock --council off --trace "Force single-model routing."

Presets set the operating posture without changing model cards:

• --preset fast: single-model, no bids, no council; uses evidence and cost pressure to prefer cheap earned routes.
• --preset balanced: default evidence/cost-aware routing.
• --preset max: bid-first and more quality-seeking; allows larger councils, one fallback, and lower cost/latency penalties without blindly forcing a council.

llm-market run \
  --provider codex \
  --config examples/codex_estimated_cost_models.json \
  --evidence outputs/current/evidence_codex_current.jsonl \
  --preset fast \
  "Summarize this diff and flag likely bugs."

live-eval calls real providers and compares the routed answer with baselines:

llm-market live-eval \
  --provider codex \
  --config examples/codex_estimated_cost_models.json \
  --evidence outputs/current/evidence_codex_current.jsonl \
  --tasks examples/codex_complex_multistep_tasks.jsonl \
  --baseline-mode external \
  --baseline-config examples/codex_complex_baselines.json \
  --provider-timeout-seconds 240 \
  --max-tokens 700 \
  --progress \
  --output outputs/archive/new_codex_run.json

Use --baseline-mode external --baseline-config ... when you want a clean comparison against direct baselines such as xhigh-for-everything and cheap fixed-single arms without direct-running every selectable candidate. Omit --baseline-mode to keep the default full-baseline behavior.

Use --best-single auto for product routing. When similar prior outcomes show that a selectable model repeatedly dominates, the router can use that model directly.

evidence-ingest turns live-eval artifacts into future routing evidence:

llm-market evidence-ingest \
  --results outputs/archive/<new-run>.json \
  --output outputs/current/evidence_codex_current.jsonl

Evidence can include prompt text, run identity, latency, route metadata, and model outcomes. Redact or keep private any eval artifact that contains customer, repo, or personal task text before publishing an evidence store.

Repo Map

• src/llm_market/analysis.py: prompt profiling.
• src/llm_market/routing.py: utility, cost, latency, diversity, and judge selection.
• src/llm_market/bidding.py: model self-bids for market routing.
• src/llm_market/evidence.py: prior-result observations and evidence deltas.
• src/llm_market/engine.py: route execution and fallback behavior.
• src/llm_market/judge.py: heuristic judge and LLM judge.
• src/llm_market/clients.py: OpenRouter/OpenAI-compatible client and Codex CLI client.
• examples/codex_estimated_cost_models.json: cost-aware Codex model cards.
• examples/codex_complex_multistep_tasks.jsonl: harder rubric-graded Codex eval tasks.
• examples/codex_complex_baselines.json: direct xhigh and cheap-single baselines for complex-task evals.
• outputs/current/codex_cost_smoke_summary.json: tracked summary for the README result.
• outputs/current: current evidence packet and active evidence JSONL.
• outputs/archive: archived run-specific reports behind the current packet.
• skills/llm-market: Codex skill wrapper and integration notes.

OpenRouter Fusion Relationship

OpenRouter Fusion made the council primitive legible: ask multiple models, have a judge compare answers, then produce one response. LLM Market keeps that primitive open and adds the route policy around it: when to pay for a council, when to ask for bids, when a cheap model is enough, when prior evidence should promote one model, and what trace data should be saved for the next prompt.

Scope

• Codex cost comparisons use estimated relative costs because Codex CLI does not expose per-call billing.
• Provider-specific cache pricing, KV-cache accounting, and cache-hit discounts are future cost-model work.
• The included live result is a small complex-task eval. Run your own task families before trusting any routing policy in production.
• Model cards need current prices, reliability estimates, and observed evidence to stay useful.

Third-Party Notices

See THIRD_PARTY_NOTICES.md.