Here it is on a real spec — examples/task-manager, a small CRUD web app, is included in this repo so you can run it yourself:

$ specjudge examples/task-manager

                   Model comparison (SpecJudge) - judge: devstral-small-2                   
┏━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃    ┃ Model                         ┃ Rating   ┃ Price                       ┃ Priced on  ┃
┡━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ 🥇 │ Mistral Small 3.2 24B (local) │ good     │ open-source/free            │ 2026-07-20 │
│ 🥈 │ DeepSeek V4 Flash             │ good     │ 0.28 out / 0.14 in USD/1M   │ 2026-07-28 │
│ 🥉 │ Gemma 4 26B MoE               │ good     │ 0.33 out / 0.06 in USD/1M   │ 2026-07-20 │
│    │ Claude Fable 5                │ overkill │ 50.00 out / 10.00 in USD/1M │ 2026-07-28 │
└────┴───────────────────────────────┴──────────┴─────────────────────────────┴────────────┘

🥇 Gold: Mistral Small 3.2 24B (local)
   Mistral Small 3.2 24B (local) is a good fit for this project. Right-sized: capability 
matches demand exactly in every dimension. Deciding dimension: 'reasoning' (demand=medium, 
capability=medium -> good).
🥈 Silver: DeepSeek V4 Flash
   DeepSeek V4 Flash is a good fit for this project. Right-sized: capability matches demand 
exactly in every dimension. Deciding dimension: 'reasoning' (demand=medium, 
capability=medium -> good).
🥉 Bronze: Gemma 4 26B MoE
   Gemma 4 26B MoE is a good fit for this project. Right-sized: capability matches demand 
exactly in every dimension. Deciding dimension: 'reasoning' (demand=medium, 
capability=medium -> good).

Abridged: the real run scores all 37 models in the catalog and lists every one of them. Prices and ratings above are generated from data/models.yaml, so they cannot go stale silently.

Mistral Small 3.2 24B (local) is right-sized for this project and costs nothing per token — it runs on your own hardware. Claude Fable 5, the priciest option this project does not need, bills $50.00/M output, $10.00/M input. If you would rather not self-host, the cheapest hosted model that still fits is DeepSeek V4 Flash at $0.28/M output, $0.14/M input — 179× cheaper on output than reaching for the frontier.

The demand profile was estimated once by a local judge and pinned, so this example is reproducible without Ollama; everything else is generated from the live catalog.

Add --open for a visual matrix in your browser, filterable by model family and by open source.

The problem

You've finished defining a project with Spec-Driven Development. Now you have to pick a model to implement it. Two expensive mistakes are waiting:

SpecJudge puts that decision at the one moment it's cheapest to make it: the specs exist, but nothing has been spent yet. And it grounds the answer in your project, not a generic benchmark.

Quick start

Requirements: Python 3.11+, uv, and Ollama 0.5.0+ with at least one local model. The version matters: asking the judge for cited evidence needs structured outputs, added in 0.5.0. On an older Ollama the run stops with an error saying so — and how to proceed without citations.

ollama pull llama3.1:8b            # a judge to evaluate your project
uv tool install specjudge
specjudge /path/to/your/project

That's it. On first run SpecJudge lists your local models, asks which one to use as the judge, and remembers your choice.

# Ephemeral run, no install
uvx specjudge /path/to/your/project

# pipx works too
pipx install specjudge

# From source
git clone https://github.com/JoaquinRuiz/SpecJudge.git && cd SpecJudge
uv sync --extra dev
uv run specjudge --help

📺 Prefer to watch? There's a walkthrough on YouTube — in Spanish.

Roadmap

What's being worked on, what's next, and where help is wanted lives in a single public roadmap issue — kept current there rather than duplicated here, where it would rot.

The most useful contribution is keeping the model catalog accurate: models and prices change every few weeks, and updating them takes zero Python. Several issues tagged good first issue don't require touching the code either.

Privacy

The judge runs on your machine through Ollama. Your specs — your business logic, your design decisions — never touch a third-party service, and deciding which model to buy costs you nothing in API calls. The browser report is a self-contained HTML file that loads nothing from the network.

This isn't a nice-to-have; it's Principle I of the project constitution.

How it works

  your project                  local judge                  model catalog
┌───────────────┐            ┌───────────────┐            ┌────────────────┐
│ constitution  │            │  Ollama, on   │            │  models.yaml   │
│ spec          │──────────► │ your machine  │──────────► │  (community-   │
│ tasks         │  how hard  │  estimates    │  compare   │   maintained)  │
└───────────────┘  is this?  │  the demand   │  capability└────────────────┘
                             └───────────────┘  vs demand         │
                                                                  ▼
                                                          🥇 🥈 🥉 podium

The judge estimates how demanding your project is across a few dimensions. Declarative rules in rating-rules.yaml cross that demand against each model's declared capability. The best fit wins — no black box, and every verdict comes with a reason you can read.

And the reason has to be more than well written. For each dimension the judge must cite the fragment of your spec that supports its rating, and SpecJudge checks that fragment really exists in the text it was given:

Evidence: 3 of 3 dimensions grounded in cited evidence
   reasoning: medium — cites S:FR-004
   size: medium — cites T:T009
   domain_specialization: low — cites S:FR-001

A judge that invents a citation has its whole assessment rejected, and one that cannot ground a dimension answers unsupported rather than guessing — that dimension then leaves the calculation instead of quietly counting as easy work.

This checks grounding, not relevance: a judge can cite something that exists but doesn't really support the level, and no automatic check catches that. It is still a great deal more than a paragraph that sounds convincing.

Reading the output

Every model gets a rating on a fixed scale:

The podium ranks models by how well they fit your project's complexity — neither under-capable nor more powerful than needed.

Price does not decide the ranking. It's shown for every model so you can see what each option costs, and it only breaks ties between models that fit equally well. A cheaper model never wins over one that fits better — recommending something that can't do the job is the most expensive mistake of all. Right-sizing is what keeps you from overpaying.

If no model in the catalog is capable enough, SpecJudge says so instead of recommending one anyway. Same when your project is too thinly specified to judge — see Data states below.

specjudge [PROJECT_PATH] [OPTIONS]

Using it from your own code

--json emits a payload with its own versioned schema, which you can fetch without cloning anything:

specjudge . --judge llama3.1:8b --json      # the result
specjudge --print-schema                     # the contract it conforms to

That covers any language. From Python there is a small documented entry point:

from specjudge import api

comparison = api.analyze("path/to/project", judge_model="llama3.1:8b")
print(comparison.best_choice)

Everything exported from specjudge.api is covered by semantic versioning; everything else under specjudge.* is internal and changes without notice. The full surface, the schema rules and what is deliberately not promised are in docs/api.md.

Contributing

Models and prices change every few weeks. That's the whole reason this project needs you.

The most valuable contribution requires zero Python — SpecJudge keeps everything volatile in human-readable data files, deliberately separate from the code. Adding a model is one YAML block:

- id: my-new-model
  name: My New Model
  family: MyFamily         # groups it in the report's filters (Claude, GPT, Qwen...)
  open_source: false       # open weights you can download and self-host?
  provider: SomeAI
  capabilities:
    reasoning: high        # low | medium | high | top
    size: high
    domain_specialization: medium
  price:
    input_per_million: 1.50
    output_per_million: 6.00
    currency: USD
    pricing_date: 2026-07-28   # required — makes freshness verifiable

Open data/models.yaml, add it, send the PR. Contract tests validate the schema automatically, so a mistake can't slip through unnoticed.

Good first contributions

Disagree with a rating? That's a contribution, not a complaint — the catalog is opinion made inspectable. Open a PR with your reasoning.

Before you send a PR

uv run pytest        # full suite — Ollama is mocked, no local models needed
uv run ruff check .

See CONTRIBUTING.md for the full schemas and project governance.

What won't be merged

Contributions that add mandatory paid dependencies to the core. A tool whose purpose is to save you money must not require money to run. That's Principle III, and it isn't negotiable.

About the author

Joaquín Ruiz — jokiruiz.com · youtube.com/@jokioki

📗 Del vibe coding al Spec-Driven Development 📙 El motor de la Inteligencia Artificial 📘 Programar con Inteligencia Artificial 📙 Explora la Inteligencia Artificial

License

MIT © Joaquín Ruiz