cje-eval 0.2.8

Causal Judge Evaluation - Unbiased LLM evaluation framework

CJE - Causal Judge Evaluation

Your LLM judge scores are lying. CJE calibrates them to what actually matters.

---

Quick Start

pip install cje-eval

from cje import analyze_dataset

# Point to your response files (one JSONL per policy)
results = analyze_dataset(fresh_draws_dir="data/responses/")

# Get calibrated estimates with valid confidence intervals
results.plot_estimates(
    policy_labels={"prompt_v1": "Conversational tone", ...},
    save_path="ranking.png"
)

Data format (one JSONL file per policy):

{"prompt_id": "1", "judge_score": 0.85, "oracle_label": 0.9}
{"prompt_id": "2", "judge_score": 0.72}

Only 5-25% of samples need oracle labels. CJE learns the judge→oracle mapping and applies it everywhere.

---

Why You Need This

Raw LLM judge scores suffer from systematic biases that make your metrics unreliable:

• Preference inversion: Higher scores often predict lower real-world quality
• Invalid confidence intervals: Standard error bars yield 0% coverage
• Scale arbitrariness: Is "4.2" actually better than "4.0"?

CJE fixes this by treating your judge as a sensor that must be calibrated against ground truth.

Read the full explanation →

---

The Proof

We benchmarked 14 estimators on 5,000 real Chatbot Arena prompts using GPT-5 as oracle:

Illustrative output comparing prompt variants

Method	Result
Raw Judges	0% CI coverage — error bars were mathematical lies
CJE (Direct + Two-Stage)	99% ranking accuracy with just 5% oracle labels

Cost savings: CJE achieves oracle-quality rankings at 14× lower cost by calibrating a cheap judge (~250 labels) instead of labeling everything.

Read the full Arena Experiment → ・ Paper (Zenodo)

---

Monitoring Calibration Over Time

Calibration can drift. Periodically verify it still holds with a small probe:

from cje import analyze_dataset
from cje.diagnostics import audit_transportability

# results.calibrator is automatically fitted during analysis
results = analyze_dataset(fresh_draws_dir="responses/")

# Check if calibration still works on this week's data (40-60 oracle labels)
diag = audit_transportability(results.calibrator, this_week_samples)
print(diag.summary())
# Transport: PASS | N=48 | δ̂: +0.007 (CI: [-0.05, +0.06])

PASS means your calibration is still valid. FAIL means something changed — investigate or recalibrate.

---

Try It Now

Open the interactive tutorial in Google Colab →

Walk through a complete example: compare prompt variants, check if calibration transfers, inspect what's fooling the judge, and monitor drift over time. No setup required.

---

Documentation

Technical Guides

• Calibration Methods — AutoCal-R, isotonic regression, two-stage
• Diagnostics System — Uncertainty quantification, transportability
• Estimators — Direct, IPS, DR implementations
• Interface/API — analyze_dataset implementation

Examples & Data

• Examples Folder — Working code samples
• Arena Sample Data — Real-world test data

---

Development

git clone https://github.com/cimo-labs/cje.git
cd cje && poetry install && make test

Support

• Issues
• Discussions

License

MIT — See LICENSE for details.