Skip to main content

PeerEval

made-with-python

PeerEval ranks large language models by having them judge each other. Every model answers the same questions; then every model compares pairs of anonymous answers. PeerEval turns those pairwise verdicts into a leaderboard using PRD Peer Rank, which gives a judge more weight when its own answers are rated well by its peers.

Leaderboard: https://peereval-ai.github.io

Install

pip install peereval            # scoring only, no dependencies
pip install "peereval[judge]"   # also generate judgments through an API

From source:

git clone https://github.com/peereval-ai/peer_rank_tool.git
cd peer_rank_tool && pip install -e ".[judge]"

Two steps

solver answers ──(peereval judge)──> pairwise judgments ──(peereval score)──> leaderboard

If you already have pairwise judgments, you only need peereval score.

Scoring judgments

Python

import peereval

board = peereval.score("examples/judgments.jsonl")

print(board.to_markdown())                 # overall + per-dataset tables
overall = board.overall
overall.ranking()                          # models, best first
overall.prd                                # PRD score per model
overall.judge_weights                      # PRD weight of each model as a judge
overall.self_bias["gpt-6-astra"]           # {"self", "others", "bias_pp"}
overall.pairwise["claude-opus-5-5"]["glm-5.3"]   # share of comparisons won

board = peereval.score("examples/judgments.jsonl", bootstrap=1000)
board.overall.ci["prd"]["glm-5.3"]          # [low, high], 95% interval
board.overall.rank_ub                      # best rank each model could have

Parameters of peereval.score:

  • judgments: path(s) to judgment JSONL files, or a list of judgment dicts.
  • models: contestants to keep. Default: every model that appears in a pair. Judges do not have to be contestants: verdicts from outside judges count toward win rate, Bradley-Terry and the pairwise matrix, but PRD weights only go to judges that are also contestants. If no judge is a contestant, PRD is left empty and models are ranked by win rate.
  • by_dataset: also score each dataset separately. Default True.
  • complete_only: keep only questions that have every verdict (all pairs × all judges × both orders). Default False.
  • bootstrap: number of bootstrap samples for 95% confidence intervals and Rank (UB), e.g. 1000. Default 0 (off).
  • seed: random seed for the bootstrap. Default 0.

Command line

peereval score judgments/*.jsonl --bootstrap 1000 --json results.json --csv leaderboard.csv

Options: --models, --no-by-dataset, --complete-only, --bootstrap N, --seed.

Example output on examples/judgments.jsonl (3 models, 3 questions):

Rank Model PRD score Judge weight Win rate Bradley-Terry Self-bias (pp)
1 claude-opus-5-5 0.754 0.546 0.639 +0.341 +22.9
2 glm-5.3 0.644 0.454 0.458 -0.154 +31.2
3 gpt-6-astra 0.102 0.000 0.403 -0.306 +89.6

Generating judgments

peereval judge takes solver answers, asks every judge to compare every pair of answers to the same question in both display orders, and writes judgments in the format peereval score reads. Judges are called through any OpenAI-compatible endpoint (OpenAI, OpenRouter, vLLM, SGLang, ...).

peereval judge answers/*.jsonl --judges judges.json --out judgments.jsonl --concurrency 8
peereval score judgments.jsonl

The run is resumable: rerunning skips verdicts already in --out and retries failed ones.

judges.json is a list of judges (see examples/judges.json; the model strings there are examples, so replace them with your provider's ids):

[
  {"name": "gpt-6-astra", "model": "openai/gpt-6-astra", "base_url": "https://openrouter.ai/api/v1", "api_key_env": "OPENROUTER_API_KEY"},
  {"name": "qwen3.8-27b", "model": "Qwen/Qwen3.8-27B", "base_url": "http://localhost:8000/v1",
   "extra_body": {"chat_template_kwargs": {"enable_thinking": false}}}
]
  • name must match the model field of that model's solver answers, so the judge is linked to its own answers (needed for PRD weights and self-bias).
  • extra_body is passed to the endpoint as is, e.g. to turn off a local model's thinking.
  • max_tokens (default 8192) and temperature (default 0) can also be set. OpenAI reasoning models need "max_tokens_param": "max_completion_tokens" and "temperature": null.

From Python:

from peereval import JudgeConfig, run_judging

judges = [JudgeConfig(name="gpt-6-astra", model="openai/gpt-6-astra",
                      base_url="https://openrouter.ai/api/v1", api_key_env="OPENROUTER_API_KEY")]
run_judging("answers.jsonl", judges, "judgments.jsonl")

Answers should contain only the final answer. If a model returns its reasoning inside the answer, the judges see it. <think>...</think> blocks are removed before judging by default (--keep-think turns this off), but reasoning written without tags is not.

Data format

Solver answers (input to peereval judge)

One JSON object per line:

{"dataset": "arena_hard_v2", "question_id": "0f93825f88454fb6", "model": "glm-5.3",
 "question": "…", "response": "…"}

Only questions answered by every model are judged.

Judgments (output of peereval judge, input to peereval score)

{"dataset": "arena_hard_v2", "question_id": "0f93825f88454fb6",
 "solver_left": "claude-opus-5-5", "solver_right": "glm-5.3",
 "response_a_model": "glm-5.3", "response_b_model": "claude-opus-5-5",
 "judge_model": "gpt-6-astra", "position": "BA",
 "judgment": {"winner": "A", "confidence": 0.8, "reason": "…"}}
  • winner is "A", "B" or "tie", and refers to the displayed order: "A" means the answer by response_a_model won.
  • solver_left / solver_right name the pair and stay fixed across the two display orders; position is "AB" when solver_left was shown first.
  • If position and response_a_model disagree, PeerEval trusts response_a_model (what the judge actually saw) and prints a warning.
  • Rows with an error field or without a valid winner are skipped. Extra keys are ignored.

Metrics

Metric What it is
PRD score Peer Rank. All judges start with equal weight. A model's score is the weighted average of the judges' win rates for it. The scores are scaled to 0–1 and normalized to sum to 1, and become the judges' new weights; this repeats until the weights stop changing.
Judge weight The final PRD weight of each model as a judge (sums to 1). Because of the 0–1 scaling, the lowest-scoring model always gets weight 0.
Win rate Average result over all of a model's comparisons (win 1, tie 0.5, loss 0), every judge counted equally.
Bradley-Terry Log-strength from a Bradley-Terry model fitted to the same verdicts (ties count half).
Self-bias How often a model prefers its own answer when it is the judge, minus how often the other judges prefer that model's answer, in percentage points.
Pairwise matrix pairwise[a][b]: share of comparisons in which a beats b.
95% CI Bootstrap confidence interval: questions are resampled with replacement (within each dataset) and every score is recomputed; the interval covers the middle 95% of the results. Available for PRD score, win rate and Bradley-Terry.
Rank (UB) 1 + the number of models whose PRD interval lies entirely above this model's. Models with the same Rank (UB) are not clearly separated.
Position bias Per judge: share of decisive verdicts won by the answer shown first, and how often the verdict is unchanged when the two answers are swapped.

License

MIT

Metadata

Release files for peereval 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for peereval 0.1.0
File Size Uploaded Provenance Actions
peereval-0.1.0.tar.gz 27.6 kB Without provenance Details for peereval-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for peereval 0.1.0
File Interpreter ABI Platform Actions
peereval-0.1.0-py3-none-any.whl Python 3 none any Details for peereval-0.1.0-py3-none-any.whl

Total release size: 47.2 kB

Release files / peereval-0.1.0.tar.gz

Download URL peereval-0.1.0.tar.gz
Size 27.6 kB
Tags Source
SHA-256 checksum
How to use checksums
cb5e3f2e483c37cf14099f2f4db9adf9c70ab8f49ceec68bac174e1565dd803b
BLAKE2b-256 checksum
How to use checksums
395b09e05dc72bb4bf1ee12c42e5f39ad9f1ab8950ad24d6f49ce75858014e6a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.16

Release files / peereval-0.1.0-py3-none-any.whl

Download URL peereval-0.1.0-py3-none-any.whl
Size 19.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
e11b9ece0a98734b3698d22cf371d234151e5d4f3470d6dddd2585de497644d8
BLAKE2b-256 checksum
How to use checksums
96ba5d21ba762684fd8e7907596baf4b05637ba3880f9716654ca8b35e4ebab6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.16

Security for peereval 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page