cje-eval 0.2.23

Causal Judge Evaluation - Unbiased LLM evaluation framework

CJE - Causal Judge Evaluation

Your LLM judge scores are noisy and weakly calibrated. CJE calibrates them to what actually matters.

We ran 16,000+ tests on Chatbot Arena data. Without calibration, 95% confidence intervals captured the true value 0% of the time. CJE restores reliable uncertainty and ranking decisions with a small oracle slice.

---

Quick Start

pip install cje-eval
# Optional (for plotting):
pip install "cje-eval[viz]"

from cje import analyze_dataset

# Compare policies on the same evaluation prompts
# Structure: { policy_name: [samples] }
# Each sample needs: prompt_id, judge_score
# Optional: oracle_label (human ground truth) on 5-25% of samples

results = analyze_dataset(
    fresh_draws_data={
        "gpt-4o": [
            {"prompt_id": "eval_001", "judge_score": 0.85, "oracle_label": 0.9},
            {"prompt_id": "eval_002", "judge_score": 0.72, "oracle_label": 0.7},
            {"prompt_id": "eval_003", "judge_score": 0.68},
            {"prompt_id": "eval_004", "judge_score": 0.79},
        ],
        "claude-sonnet": [
            {"prompt_id": "eval_001", "judge_score": 0.78, "oracle_label": 0.82},
            {"prompt_id": "eval_002", "judge_score": 0.81, "oracle_label": 0.79},
            {"prompt_id": "eval_003", "judge_score": 0.75},
            {"prompt_id": "eval_004", "judge_score": 0.83},
        ],
    }
)

# Or from files: analyze_dataset(fresh_draws_dir="responses/")

# Optional: plotting requires matplotlib (pip install "cje-eval[viz]")
results.plot_estimates(save_path="ranking.png")

CJE learns the judge→oracle mapping from the labeled samples and applies it everywhere. Notation used in docs/playbook: S = judge score signal (judge_score), Y = oracle target label (oracle_label).

Default recommendation: use Direct mode (fresh_draws_*) for most evaluation workflows. Advanced note: IPS/DR variants are supported for counterfactual OPE, but are not part of the default operational loop.

---

Label Compatibility

CJE automatically handles different label scales without manual preprocessing:

# 0-100 scores work automatically
results = analyze_dataset(
    fresh_draws_data={
        "gpt-4o": [
            {"prompt_id": "1", "judge_score": 85, "oracle_label": 78},  # 0-100 scale
            {"prompt_id": "2", "judge_score": 72, "oracle_label": 65},
        ],
    }
)

# Results are returned in YOUR scale (0-100), not [0,1]
print(results.estimates[0])  # → 73.5 (not 0.735)

Supported: [0,1], 0-100, Likert 1-5, or any bounded range. If values are already in [0,1], no transformation is applied.

---

Why You Need This

LLM-as-judge gives you rankings. CJE gives you certainty.

Without calibration, you know prompt A scored higher than B—but you don't know:

• Is the difference real or noise?
• How big is the improvement, actually?
• Have I tested enough samples?
• Will this hold next week?

CJE answers all of these. Label a small slice with your oracle (human raters, latest SOTA model or AI agent, downstream metric). CJE learns the calibration and applies it everywhere—giving you trustworthy magnitudes, valid confidence intervals, and drift detection.

The result: Make decisions faster, spend less on labeling, and defend your conclusions with real statistics.

Read the full explanation →

---

The Results

We tested on 5,000 Chatbot Arena prompts with GPT-5 as the oracle (ground truth) and GPT-4.1-nano as the cheap judge:

CJE achieves 99% ranking accuracy using only 5% oracle labels—matching full-oracle performance at 14× lower cost.

Label ~250 samples with your oracle (human raters, downstream KPIs, expensive model). CJE learns the judge→oracle mapping and applies it to everything else. Without calibration, error bars contained the true value 0% of the time. With CJE: ~95%.

Already using an expensive model for evals? Switch to a 10-30× cheaper judge + CJE calibration. Same accuracy, fraction of the inference cost.

Example output with simulated data (not real model benchmarks)

Read the full Arena Experiment →

---

Monitoring Calibration Over Time

Calibration can drift. Periodically verify it still holds with a small probe:

from cje import analyze_dataset
from cje.diagnostics import audit_transportability

# results.calibrator is automatically fitted during analysis
results = analyze_dataset(fresh_draws_dir="responses/")

# Check if calibration still works on this week's data (50+ oracle labels)
diag = audit_transportability(results.calibrator, this_week_samples)
print(diag.summary())
# Transport: PASS | Group: ... | N=50 | δ̂: +0.012 (CI: [-0.008, +0.032])

PASS means your calibration is still valid. FAIL means something changed — investigate or recalibrate.

---

Try It Now

Open the interactive tutorial in Google Colab →

Walk through a complete example: compare policies, check if calibration transfers, inspect what's fooling the judge, and monitor drift over time. No setup required.

---

Documentation

Planning sample sizes? Use pilot data to optimize your evaluation budget:

Video Walkthroughs

• CJE Technical Walkthrough — Pipeline deep dive: calibration, evaluation, and transport auditing
• CJE in 3 Minutes — Quick intro: why raw judge scores mislead and how CJE fixes it

Technical Guides

• Operational Playbook — End-to-end runbook: audits, failed-audit correction, and label budgeting
• Drift Correction Research Note — Offset vs EIF-style correction vs refit after audit drift
• Calibration Methods — AutoCal-R, isotonic regression, two-stage
• Diagnostics System — Uncertainty quantification, transportability
• Estimators — Estimator internals (advanced, including IPS/DR)
• Interface/API — analyze_dataset implementation
• Experiments — repo-only simulation studies (not shipped in PyPI)

Bridges (Promptfoo / TruLens / LangSmith / OpenCompass → CJE)

Note: these bridge converters are repo scripts (they are not installed with pip install cje-eval). Clone the repo to use them:

git clone https://github.com/cimo-labs/cje.git
cd cje

If you already run evals in Promptfoo, TruLens, LangSmith, or OpenCompass, you can convert those outputs into CJE’s fresh_draws_data format.

# Promptfoo
python3 scripts/cje_bridges/convert.py promptfoo results.json \
  --out cje_fresh_draws_data.json \
  --label-template oracle_label_template.csv

# TruLens (install first: pip install trulens)
python3 scripts/cje_bridges/convert.py trulens \
  --database-url sqlite:///default.sqlite \
  --judge-col "Answer Relevance" \
  --out cje_fresh_draws_data.json \
  --label-template oracle_label_template.csv

# LangSmith (install first: pip install langsmith; set LANGSMITH_API_KEY)
python3 scripts/cje_bridges/convert.py langsmith \
  --project "my_model_a_project" \
  --project "my_model_b_project" \
  --feedback-key "correctness" \
  --out cje_fresh_draws_data.json \
  --label-template oracle_label_template.csv

# OpenCompass (LLM-as-judge; run OpenCompass with --dump-eval-details)
python3 scripts/cje_bridges/convert.py opencompass path/to/opencompass_results.json \
  --out cje_fresh_draws_data.json \
  --label-template oracle_label_template.csv

After you label an oracle slice, re-run the converter to populate oracle_label:

• Promptfoo/TruLens/OpenCompass: pass --oracle-labels <your_labeled_csv_or_jsonl>
• LangSmith: if labels are stored in LangSmith as feedback, pass --oracle-feedback-key <key>

See: scripts/cje_bridges/README.md

Examples & Data

• Examples Folder — Working code samples
• Arena Sample Data — Real-world test data

---

Development

git clone https://github.com/cimo-labs/cje.git
cd cje && poetry install && make test

Support

• Issues

Citation

If you use CJE in your research, please cite:

@misc{landesberg2025causaljudgeevaluationcalibrated,
  title={Causal Judge Evaluation: Calibrated Surrogate Metrics for LLM Systems},
  author={Eddie Landesberg},
  year={2025},
  eprint={2512.11150},
  archivePrefix={arXiv},
  primaryClass={stat.ME},
  url={https://arxiv.org/abs/2512.11150},
}

License

MIT — See LICENSE for details.