pairsort

Rank anything with AI judges: reliably, cheaply, and with honest confidence.

Ask a model to score 100 things from 1 to 10 and you get noise on a drifting scale. Ask it "which of these two is better?" and you get its best judgment. pairsort asks many of those small questions, cancels the model's biases, and couples the answers into one ranking with probabilities, using the pairwise-coupling rule of Price, Knerr, Personnaz & Dreyfus (NeurIPS 1994).

• More reliable than any single judgment. Coupling many pairwise answers fixes the contradictions a judge makes (A > B > C > A) and averages out its noise and position bias. Measured against explicit ground truth in eight evals.
• Cheap: not all-vs-all. An active schedule asks only the informative pairs and stops when the ranking settles; 100 items were ranked from 8% of the possible pairs. With Jev as the judge, 4,800 comparisons cost $0.11.
• Probabilities, not vibes. Every item gets a posterior, every pair a calibrated P(A beats B), and pairsort abstains when the top two are too close to call.
• Any judge. TypeSafe's Jev via OpenRouter (default), any OpenRouter LLM via token logprobs, open Jev models, or your own: judges.
• Auditable. Every judgment, in both orders, lands in a JSON audit log.

Try it in 60 seconds

pip install pairsort                       # PyPI name; you `import pairsort` and run `pairsort` (or `pairsort`)
uvx pairsort demo                          # or run the CLI without installing anything
export OPENROUTER_API_KEY=sk-or-...        # or skip it and bring your own judge function

import pairsort

ideas = ["A CLI that turns any CSV into a chart", "A to-do app on a blockchain", "A tool that ranks PRs by urgency"]

result = pairsort.sort(ideas, "Which side project would developers find most useful?")
result.best        # 'A tool that ranks PRs by urgency'   (your own objects back, best first)
result.top(2)      # the two best
result.scores      # {id: probability of being the best}

pairsort.sort(ideas, useful="Which is more useful?", easy="Which is easier to build?")       # blend questions
pairsort.compare("draft A", "draft B", "Which is clearer?")                                    # -> P(A is better)
pairsort.sort(ideas, "Which is shorter?", judge=lambda q, a, b: len(a) < len(b))               # any function is a judge

pairsort sort ideas.txt "Which idea has more impact?"          # from a file (`uvx pairsort sort ...` works too)
cat ideas.txt | pairsort sort - "Which is funnier?" --top 3     # from stdin
pairsort compare "draft A" "draft B" "Which is clearer?"

Nothing is hidden behind the easy path: budgets, active/referee schedules, meta-judges, calibration profiles and any backend are all keywords on pairsort.sort or steps on the builder (pairsort.sorter().by(...).budget(60).meta()). Quickstart (runs in ~5 s: python examples/quickstart.py) · Usage & CLI

Summary Showdown

100 of OpenRouter's most-used models each summarized the 1994 paper this library implements. pairsort ranked them on six questions from 400 of 4,950 possible pairs (8%) with a jury of 4 AI judges. Current top 3: GPT-6 Astra, Claude Opus 5, Nemotron 3 Ultra.

Ground truth: none — nobody can say which summary is truly best. The ranking is the AI jury's opinion; we check it against a separate LLM grader (Claude Sonnet 5 + a key-fact rubric), which the verifiable eval shows is itself accurate on exact counts. Humans can vote on the site.

Judge the summaries yourself → · full results + method · leaderboard

Does it work?

Eight evals, each with its ground truth stated up front:

Ground truth in this figure: exact counts built into fictional documents; recount them with python examples/verifiable_eval.py verify.

Learn more

• How it works: pairwise questions, PKPD Eq. 7, Bradley–Terry, bias guards, blending, Jev as meta-judge and referee
• Judges & backends: Jev via OpenRouter, the LLM fallback, open Jev models, the /v1/systemone shim
• Usage: CLI, input formats, Python API, calibration, a worked example
• Evals: verifiable · market · ladder · runtime · cross-lingual · weather · synthetic + papers · Summary Showdown · Humans vs judges

MIT licensed. Contributions welcome.