PeerEval
PeerEval ranks large language models by having them judge each other. Every model answers the same questions; then every model compares pairs of anonymous answers. PeerEval turns those pairwise verdicts into a leaderboard using PRD Peer Rank, which gives a judge more weight when its own answers are rated well by its peers.
Leaderboard: https://peereval-ai.github.io
Install
pip install peereval # scoring only, no dependencies
pip install "peereval[judge]" # also generate judgments through an API
From source:
git clone https://github.com/peereval-ai/peer_rank_tool.git
cd peer_rank_tool && pip install -e ".[judge]"
Two steps
solver answers ──(peereval judge)──> pairwise judgments ──(peereval score)──> leaderboard
If you already have pairwise judgments, you only need peereval score.
Scoring judgments
Python
import peereval
board = peereval.score("examples/judgments.jsonl")
print(board.to_markdown()) # overall + per-dataset tables
overall = board.overall
overall.ranking() # models, best first
overall.prd # PRD score per model
overall.judge_weights # PRD weight of each model as a judge
overall.self_bias["gpt-6-astra"] # {"self", "others", "bias_pp"}
overall.pairwise["claude-opus-5-5"]["glm-5.3"] # share of comparisons won
board = peereval.score("examples/judgments.jsonl", bootstrap=1000)
board.overall.ci["prd"]["glm-5.3"] # [low, high], 95% interval
board.overall.rank_ub # best rank each model could have
Parameters of peereval.score:
judgments: path(s) to judgment JSONL files, or a list of judgment dicts.models: contestants to keep. Default: every model that appears in a pair. Judges do not have to be contestants: verdicts from outside judges count toward win rate, Bradley-Terry and the pairwise matrix, but PRD weights only go to judges that are also contestants. If no judge is a contestant, PRD is left empty and models are ranked by win rate.by_dataset: also score eachdatasetseparately. DefaultTrue.complete_only: keep only questions that have every verdict (all pairs × all judges × both orders). DefaultFalse.bootstrap: number of bootstrap samples for 95% confidence intervals and Rank (UB), e.g.1000. Default0(off).seed: random seed for the bootstrap. Default0.
Command line
peereval score judgments/*.jsonl --bootstrap 1000 --json results.json --csv leaderboard.csv
Options: --models, --no-by-dataset, --complete-only, --bootstrap N, --seed.
Example output on examples/judgments.jsonl (3 models, 3 questions):
| Rank | Model | PRD score | Judge weight | Win rate | Bradley-Terry | Self-bias (pp) |
|---|---|---|---|---|---|---|
| 1 | claude-opus-5-5 | 0.754 | 0.546 | 0.639 | +0.341 | +22.9 |
| 2 | glm-5.3 | 0.644 | 0.454 | 0.458 | -0.154 | +31.2 |
| 3 | gpt-6-astra | 0.102 | 0.000 | 0.403 | -0.306 | +89.6 |
Generating judgments
peereval judge takes solver answers, asks every judge to compare every pair of answers to the same question in both display orders, and writes judgments in the format peereval score reads. Judges are called through any OpenAI-compatible endpoint (OpenAI, OpenRouter, vLLM, SGLang, ...).
peereval judge answers/*.jsonl --judges judges.json --out judgments.jsonl --concurrency 8
peereval score judgments.jsonl
The run is resumable: rerunning skips verdicts already in --out and retries failed ones.
judges.json is a list of judges (see examples/judges.json; the model strings there are examples, so replace them with your provider's ids):
[
{"name": "gpt-6-astra", "model": "openai/gpt-6-astra", "base_url": "https://openrouter.ai/api/v1", "api_key_env": "OPENROUTER_API_KEY"},
{"name": "qwen3.8-27b", "model": "Qwen/Qwen3.8-27B", "base_url": "http://localhost:8000/v1",
"extra_body": {"chat_template_kwargs": {"enable_thinking": false}}}
]
namemust match themodelfield of that model's solver answers, so the judge is linked to its own answers (needed for PRD weights and self-bias).extra_bodyis passed to the endpoint as is, e.g. to turn off a local model's thinking.max_tokens(default 8192) andtemperature(default 0) can also be set. OpenAI reasoning models need"max_tokens_param": "max_completion_tokens"and"temperature": null.
From Python:
from peereval import JudgeConfig, run_judging
judges = [JudgeConfig(name="gpt-6-astra", model="openai/gpt-6-astra",
base_url="https://openrouter.ai/api/v1", api_key_env="OPENROUTER_API_KEY")]
run_judging("answers.jsonl", judges, "judgments.jsonl")
Answers should contain only the final answer. If a model returns its reasoning inside the answer, the judges see it. <think>...</think> blocks are removed before judging by default (--keep-think turns this off), but reasoning written without tags is not.
Data format
Solver answers (input to peereval judge)
One JSON object per line:
{"dataset": "arena_hard_v2", "question_id": "0f93825f88454fb6", "model": "glm-5.3",
"question": "…", "response": "…"}
Only questions answered by every model are judged.
Judgments (output of peereval judge, input to peereval score)
{"dataset": "arena_hard_v2", "question_id": "0f93825f88454fb6",
"solver_left": "claude-opus-5-5", "solver_right": "glm-5.3",
"response_a_model": "glm-5.3", "response_b_model": "claude-opus-5-5",
"judge_model": "gpt-6-astra", "position": "BA",
"judgment": {"winner": "A", "confidence": 0.8, "reason": "…"}}
winneris"A","B"or"tie", and refers to the displayed order:"A"means the answer byresponse_a_modelwon.solver_left/solver_rightname the pair and stay fixed across the two display orders;positionis"AB"whensolver_leftwas shown first.- If
positionandresponse_a_modeldisagree, PeerEval trustsresponse_a_model(what the judge actually saw) and prints a warning. - Rows with an
errorfield or without a validwinnerare skipped. Extra keys are ignored.
Metrics
| Metric | What it is |
|---|---|
| PRD score | Peer Rank. All judges start with equal weight. A model's score is the weighted average of the judges' win rates for it. The scores are scaled to 0–1 and normalized to sum to 1, and become the judges' new weights; this repeats until the weights stop changing. |
| Judge weight | The final PRD weight of each model as a judge (sums to 1). Because of the 0–1 scaling, the lowest-scoring model always gets weight 0. |
| Win rate | Average result over all of a model's comparisons (win 1, tie 0.5, loss 0), every judge counted equally. |
| Bradley-Terry | Log-strength from a Bradley-Terry model fitted to the same verdicts (ties count half). |
| Self-bias | How often a model prefers its own answer when it is the judge, minus how often the other judges prefer that model's answer, in percentage points. |
| Pairwise matrix | pairwise[a][b]: share of comparisons in which a beats b. |
| 95% CI | Bootstrap confidence interval: questions are resampled with replacement (within each dataset) and every score is recomputed; the interval covers the middle 95% of the results. Available for PRD score, win rate and Bradley-Terry. |
| Rank (UB) | 1 + the number of models whose PRD interval lies entirely above this model's. Models with the same Rank (UB) are not clearly separated. |
| Position bias | Per judge: share of decisive verdicts won by the answer shown first, and how often the verdict is unchanged when the two answers are swapped. |
License
MIT
Metadata
Release files for peereval 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| peereval-0.1.0.tar.gz | 27.6 kB | Without provenance | Details for peereval-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| peereval-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for peereval-0.1.0-py3-none-any.whl |
Total release size: 47.2 kB
Release files / peereval-0.1.0.tar.gz
| Download URL | peereval-0.1.0.tar.gz |
|---|---|
| Size | 27.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
cb5e3f2e483c37cf14099f2f4db9adf9c70ab8f49ceec68bac174e1565dd803b
|
|
BLAKE2b-256 checksum How to use checksums |
395b09e05dc72bb4bf1ee12c42e5f39ad9f1ab8950ad24d6f49ce75858014e6a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.16
|
Release files / peereval-0.1.0-py3-none-any.whl
| Download URL | peereval-0.1.0-py3-none-any.whl |
|---|---|
| Size | 19.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e11b9ece0a98734b3698d22cf371d234151e5d4f3470d6dddd2585de497644d8
|
|
BLAKE2b-256 checksum How to use checksums |
96ba5d21ba762684fd8e7907596baf4b05637ba3880f9716654ca8b35e4ebab6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.16
|
Security for peereval 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.