winference 0.1.0

winference: Win rate calibration under non-transitivity.

When you run an LLM arena and report "Model A beats Model B 62% of the time," is that number calibrated? And does it still hold when your users ask different questions than your evaluation set?

If model strengths vary across task types, aggregate win rates can exhibit non-transitive preferences: A beats B, B beats C, but C beats A. Standard Bradley-Terry / Elo assumes this doesn't happen, and when it does, your calibration breaks — especially under distribution shift.

winference provides two approaches to calibrating win rates in the presence of non-transitivity, plus diagnostics to decide which one you need.

The two approaches

A) Hodge decomposition → calibrate the transitive signal

Decomposes the pairwise comparison matrix into:

• Gradient (transitive): a potential s_i per model such that the log-odds ≈ s_i − s_j. This part can be calibrated to a scalar ranking.
• Curl (cyclic): rock-paper-scissors structure that cannot be represented by any linear ranking.

Calibrate win rates from the gradient component. Report the curl fraction as the share of variance your calibration ignores.

Use when: Cycles persist even after conditioning on task category — the non-transitivity is irreducible.

B) Heterogeneous groups → calibrate per category, compose

Test whether model strengths differ across prompt categories (math, creative, coding, ...) using a likelihood-ratio test. If so, fit Bradley-Terry per category. Win rates for any target distribution are then:

P(A > B | π*) = Σ_k  π*_k · σ(θ_{A,k} − θ_{B,k})

This gives you composable calibration: swap in any target distribution without refitting.

Use when: Non-transitivity dissolves when you condition on prompt category.

Quickstart

pip install numpy scipy scikit-learn matplotlib
# Then from the repo root:
pip install -e .

from winference import (
    TournamentGraph, BradleyTerry, HodgeDecomposition,
    GroupTest, GroupCalibrator, expected_calibration_error,
)
from winference.simulate import simulate_llm_arena

# 1. Simulate (or load) arena data
data = simulate_llm_arena()
comparisons = data["comparisons"]   # list of (model_a, model_b, a_wins)
categories  = data["categories"]    # list of category labels per comparison
models      = data["models"]

# 2. Graph triage: is non-transitivity a problem?
tg = TournamentGraph(models)
for a, b, w in comparisons:
    tg.add_result(a, b, w)

print(tg.summary())
# → {'nontransitivity_index': 0.83, 'cyclic_triples': 7, ...}

# 3a. Hodge decomposition
hd = HodgeDecomposition(models)
result = hd.fit(tg.win_rate_matrix(), weights=tg.counts)
print(f"Transitive: {result.transitive_variance:.0%}")
print(f"Cyclic:     {result.cyclic_variance:.0%}")

# Calibrated win probability (transitive component only)
p = hd.transitive_win_probability("ZetaMath", "DeltaWrite")

# 3b. Group heterogeneity test
groups = sorted(set(categories))
gt = GroupTest(models, groups)
gt.fit(comparisons, categories)
print(gt.test_result())
# → {'statistic': 342.1, 'p_value': 1.2e-63, 'reject_at_05': True}

# Composable win rates
gc = GroupCalibrator(gt)
p_math_heavy = gc.win_probability(
    "ZetaMath", "DeltaWrite",
    target_distribution={"reasoning": 0.7, "creative_writing": 0.15, "coding": 0.15},
)
p_creative_heavy = gc.win_probability(
    "ZetaMath", "DeltaWrite",
    target_distribution={"reasoning": 0.15, "creative_writing": 0.7, "coding": 0.15},
)

See examples/quickstart.py for the full pipeline with calibration comparison and reliability diagrams.

The diagnostic pipeline

┌─────────────────────────────┐
│  Build tournament graph     │
│  Run Tarjan's SCC           │
└──────────┬──────────────────┘
           │
     All SCCs size 1?
      ╱           ╲
    YES            NO
     │              │
  Standard BT    ┌──┴───────────────────────┐
  is fine        │  Condition on categories  │
                 │  Check: do SCCs shrink?   │
                 └──────┬───────────────┬────┘
                    YES                  NO
                     │                    │
              ┌──────┴──────┐    ┌───────┴────────┐
              │ Per-group   │    │ Hodge decomp   │
              │ BT + LRT    │    │ calibrate grad │
              │ + compose   │    │ report curl    │
              └─────────────┘    └────────────────┘

Other sources of non-transitivity

Non-transitivity in pairwise comparisons doesn't always come from heterogeneous model strengths. Before reaching for Hodge or group decomposition, rule out simpler causes:

winference targets the modelling layer (rows 4–5 in the table above). It assumes you've already addressed measurement issues at the design/preprocessing level.

API reference

TournamentGraph

Build a directed tournament graph and compute SCC structure.

• .add_result(a, b, win) — record one comparison
• .strongly_connected_components() — Tarjan's algorithm
• .nontransitivity_index() — fraction of models in non-trivial SCCs
• .count_cyclic_triples() — count A>B>C>A cycles
• .summary() — quick diagnostic dict

BradleyTerry

Standard BT model via maximum likelihood.

• .fit(comparisons) — fit from (a, b, a_wins) triples
• .win_probability(a, b) — predicted P(a beats b)
• .strengths() — {model: θ}
• .rank() — models sorted by strength

HodgeDecomposition

Hodge decomposition of the pairwise log-odds matrix.

• .fit(W, weights) — decompose win-rate matrix
• .transitive_win_probability(a, b) — P(a>b) from gradient only
• .worst_pairs(k) — pairs with largest cyclic residual
• .summary() — variance fractions (transitive / cyclic / harmonic)

GroupTest

Likelihood-ratio test for heterogeneity across prompt groups.

• .fit(comparisons, group_labels) — fit null + per-group BT
• .test_result() — {statistic, df, p_value, reject_at_05}
• .per_group_strengths() — {group: {model: θ}}

GroupCalibrator

Composable win rates from per-group BT.

• .win_probability(a, b, target_distribution) — composite P(a>b)
• .sensitivity_analysis(a, b) — how much does P(a>b) vary with π*?

Calibration utilities

• expected_calibration_error(predicted, observed) — ECE
• brier_score(predicted, observed) — Brier score
• reliability_diagram(predicted, observed, ax=...) — reliability plot

Simulators

• simulate_transitive(...) — pure BT (no cycles)
• simulate_heterogeneous(...) — per-category strengths
• simulate_rock_paper_scissors(...) — irreducible cyclic structure
• simulate_llm_arena(...) — realistic six-model arena

References

• Jiang, X., Lim, L.-H., Yao, Y., & Ye, Y. (2011). Statistical ranking and combinatorial Hodge theory. Mathematical Programming, 127(1), 203–244.
• Dittrich, R., Hatzinger, R., & Katzenbeisser, W. (1998). Modelling the effect of subject-specific covariates in paired comparison studies. Applied Statistics, 47(4), 511–525.
• Xu, Y., Ruis, L., Rocktäschel, T., & Kirk, R. (2025). Investigating non-transitivity in LLM-as-a-Judge. ICML 2025.
• Li, X. & Li, S. (2025). Efficient inference for covariate-adjusted Bradley-Terry model with covariate shift. arXiv:2503.18256.
• Balduzzi, D., Tuyls, K., Perolat, J., & Graepel, T. (2018). Re-evaluating evaluation. NeurIPS.

License

MIT