xrtm-eval 0.3.1

The Judge/Scoring engine for XRTM.

xrtm-eval v0.3.0

The Judge for XRTM.

xrtm-eval is the scoring engine for probabilistic forecasts. It measures calibration, accuracy, and resolution using strict scoring rules.

Installation

pip install xrtm-eval

Scoring Metrics

Metric	Class	Description
Brier Score	BrierScoreEvaluator	Mean squared error (p - o)^2. Murphy decomposition into reliability/resolution/uncertainty.
ECE	ExpectedCalibrationErrorEvaluator	Expected Calibration Error via reliability binning.
Log Score	LogScoreEvaluator	Negative log-likelihood with epsilon clamping.
Dashboard	summarize_binary_forecasts()	Aggregate Brier, ECE, LogScore + calibration curve in one call.

Usage

from xrtm.eval import BrierScoreEvaluator, summarize_binary_forecasts

# Single forecast


evaluator = BrierScoreEvaluator()
score = evaluator.score(probability=0.7, ground_truth="yes")

# Batch evaluation
predictions = [(0.7, "yes"), (0.3, "no"), (0.9, "yes")]
summary = summarize_binary_forecasts(predictions)
print(f"Brier: {summary['brier_score']:.4f}")
print(f"ECE:   {summary['ece']:.4f}")

License

Apache 2.0