autojudge-evaluate

Evaluation tools for the TREC AutoJudge framework. Computes leaderboard correlations, inter-annotator agreement on qrels, leaderboard statistics, and format conversion for evaluation result files.

Installation

uv pip install autojudge-evaluate

CLI Commands

All commands are available via auto-judge-evaluate <command>.

meta-evaluate — Leaderboard correlation

Correlate predicted leaderboards against a ground-truth leaderboard.

auto-judge-evaluate meta-evaluate \
    --truth-leaderboard truth.eval.jsonl --truth-format jsonl \
    --eval-format tot -i results/*eval.txt \
    --correlation kendall --correlation spearman --correlation tauap_b \
    --truth-measure nugget_coverage --truth-measure f1 \
    --on-missing default \
    --output correlations.jsonl

Key options:

Output: One row per (Judge, TruthMeasure, EvalMeasure) with correlation values as columns.

qrel-evaluate — Inter-annotator agreement on qrels

Compare predicted relevance judgments (qrels) against truth qrels. Computes set overlap (precision, recall, F1) and agreement metrics (Cohen's Kappa, Krippendorff's Alpha, Jaccard, ARI).

auto-judge-evaluate qrel-evaluate \
    --truth-qrels official.qrels \
    --predict-qrels predicted.qrels

Key options:

Output: Per-topic table with Precision, Recall, F1, Jaccard, Kappa, Krippendorff's Alpha, ARI, plus a MEAN row.

leaderboard — Leaderboard statistics

Compute per-run statistics (mean, stderr, stdev, min, max) from leaderboard files.

auto-judge-evaluate leaderboard \
    --eval-format tot -i results/*eval.txt --sort

Key options:

Output: One row per (Judge, RunID, Measure) with Topics, Mean, Stderr, Stdev, Min, Max.

Analysis Module

Post-hoc analysis, tables, plots of meta-evaluate output. Produces correlation tables and bar plots with judge categorization.

python -m autojudge_evaluate.analysis.correlation_table \
    -d ragtime:ragtime-correlations.jsonl \
    -d rag:rag-correlations.jsonl \
    -d dragun:dragun-correlations.jsonl \
    --judges judges.yml \
    --correlation kendall \
    --truth-measure nugget_coverage \
    --format latex \
    --plot-dir plots/

Judge configuration (judges.yml) maps cryptic filenames to display names and categories, with optional plot styling:

styles:
  colors:
    pointwise: "#4A90D9"
    pairwise:  "#D94A4A"
  hatches:
    gpt-4o:    ""
    llama-3:   "//"

judges:
  my-judge-A.eval:
    name: System A
    method: pointwise     # category column
    model: gpt-4o         # category column
  my-judge-B.eval:
    name: System B
    method: pairwise
    model: llama-3

• styles.colors: maps category values to fill colors (any matplotlib color string)
• styles.hatches: maps category values to hatch patterns (//, .., xx, \\, etc). Values combine across categories.
• Color is picked from the first matching category value; hatches are combined from all matches.
• Without a styles: section, bars use a sequential grayscale fallback.
• Judges not in the YAML are excluded unless --all-judges is passed.

Key options: --format (github, latex, tsv, plain, html, pipe), --columns (correlations or measures), --summary (add mean/max rows), --aggregate (aggregate across datasets), --same THRESHOLD (highlight near-equal values).

eval-result — Format conversion and verification

Clean and convert evaluation result files.

# Convert tot to jsonl
auto-judge-evaluate eval-result data.txt -if tot -of jsonl -o data.jsonl

# Filter to specific runs and topics
auto-judge-evaluate eval-result data.txt -if tot -of jsonl -o filtered.jsonl \
    --filter-runs system_A --filter-runs system_B \
    --filter-topics topic_1

Key options:

Supported formats: