cje-eval 0.2.18

Causal Judge Evaluation - Unbiased LLM evaluation framework

CJE - Causal Judge Evaluation

Your LLM judge scores are noisy and nebulous. CJE calibrates them to what actually matters.

We ran 16,000+ tests on Chatbot Arena data. Without calibration, 95% confidence intervals captured the true value 0% of the time. With CJE: 99% ranking accuracy using just 5% oracle labels, at 14× lower cost.

---

Quick Start

pip install cje-eval

from cje import analyze_dataset

# Compare policies on the same evaluation prompts
# Structure: { policy_name: [samples] }
# Each sample needs: prompt_id, judge_score
# Optional: oracle_label (human ground truth) on 5-25% of samples

results = analyze_dataset(
    fresh_draws_data={
        "gpt-4o": [
            {"prompt_id": "eval_001", "judge_score": 0.85, "oracle_label": 0.9},
            {"prompt_id": "eval_002", "judge_score": 0.72, "oracle_label": 0.7},
            {"prompt_id": "eval_003", "judge_score": 0.68},
            {"prompt_id": "eval_004", "judge_score": 0.79},
        ],
        "claude-sonnet": [
            {"prompt_id": "eval_001", "judge_score": 0.78, "oracle_label": 0.82},
            {"prompt_id": "eval_002", "judge_score": 0.81, "oracle_label": 0.79},
            {"prompt_id": "eval_003", "judge_score": 0.75},
            {"prompt_id": "eval_004", "judge_score": 0.83},
        ],
    }
)

# Or from files: analyze_dataset(fresh_draws_dir="responses/")

results.plot_estimates(save_path="ranking.png")

CJE learns the judge→oracle mapping from the labeled samples and applies it everywhere.

---

Why You Need This

LLM-as-judge gives you rankings. CJE gives you certainty.

Without calibration, you know prompt A scored higher than B—but you don't know:

• Is the difference real or noise?
• How big is the improvement, actually?
• Have I tested enough samples?
• Will this hold next week?

CJE answers all of these. Label 5% of samples with your oracle (human raters, latest SOTA model or AI agent, downstream metric). CJE learns the calibration and applies it everywhere—giving you trustworthy magnitudes, valid confidence intervals, and drift detection.

The result: Make decisions faster, spend less on labeling, and defend your conclusions with real statistics.

Read the full explanation →

---

The Results

We tested on 5,000 Chatbot Arena prompts with GPT-5 as the oracle (ground truth) and GPT-4.1-nano as the cheap judge:

CJE achieves 99% ranking accuracy using only 5% oracle labels—matching full-oracle performance at 14× lower cost.

Label ~250 samples with your oracle (human raters, downstream KPIs, expensive model). CJE learns the judge→oracle mapping and applies it to everything else. Without calibration, error bars contained the true value 0% of the time. With CJE: ~95%.

Already using an expensive model for evals? Switch to a 10-30× cheaper judge + CJE calibration. Same accuracy, fraction of the inference cost.

Example output with simulated data (not real model benchmarks)

Read the full Arena Experiment →

---

Monitoring Calibration Over Time

Calibration can drift. Periodically verify it still holds with a small probe:

from cje import analyze_dataset
from cje.diagnostics import audit_transportability

# results.calibrator is automatically fitted during analysis
results = analyze_dataset(fresh_draws_dir="responses/")

# Check if calibration still works on this week's data (50+ oracle labels)
diag = audit_transportability(results.calibrator, this_week_samples)
print(diag.summary())
# Status: PASS | Samples: 48 | Mean error: +0.007 (CI: -0.05 to +0.06)

PASS means your calibration is still valid. FAIL means something changed — investigate or recalibrate.

---

Try It Now

Open the interactive tutorial in Google Colab →

Walk through a complete example: compare policies, check if calibration transfers, inspect what's fooling the judge, and monitor drift over time. No setup required.

---

Documentation

Planning sample sizes? Use pilot data to optimize your evaluation budget:

Video Walkthroughs

• CJE Technical Walkthrough — Pipeline deep dive: calibration, evaluation, and transport auditing
• CJE in 3 Minutes — Quick intro: why raw judge scores mislead and how CJE fixes it

Technical Guides

• Calibration Methods — AutoCal-R, isotonic regression, two-stage
• Diagnostics System — Uncertainty quantification, transportability
• Estimators — Direct, IPS, DR implementations
• Interface/API — analyze_dataset implementation

Examples & Data

• Examples Folder — Working code samples
• Arena Sample Data — Real-world test data

---

Development

git clone https://github.com/cimo-labs/cje.git
cd cje && poetry install && make test

Support

• Issues

Citation

If you use CJE in your research, please cite:

@misc{landesberg2025causaljudgeevaluationcalibrated,
  title={Causal Judge Evaluation: Calibrated Surrogate Metrics for LLM Systems},
  author={Eddie Landesberg},
  year={2025},
  eprint={2512.11150},
  archivePrefix={arXiv},
  primaryClass={stat.ME},
  url={https://arxiv.org/abs/2512.11150},
}

License

MIT — See LICENSE for details.