14 projects
judgekit
Bias probes for LLM judges, each with a bootstrap confidence interval: position, verbosity, self-preference, calibration and stability.
abkit
Sample-ratio, peeking, multiple-testing and winner's-curse checks for experiment readouts
abeval
A/B-test statistics for LLM evals: error bars, paired comparisons, and sample-size planning
judgepanel
Estimate LLM-judge accuracy without gold labels and aggregate judge panels: Dawid-Skene EM, agreement statistics, bootstrap uncertainty
explainkit
Explain the classifier, then test the explanation: word-level attributions, token selection, prompt steering, replacement probes and faithfulness metrics for any text classifier.
trajectory-judge
Measuring what LLM judges miss when an agent reaches the right answer the wrong way
retainkit
Context and memory policies for LLM agents, scored by the evidence that survives the token budget
raterkit
Reliability, rogue-rater, drift and leakage diagnostics for labelled data
arenakit
Bradley-Terry leaderboards with simultaneous confidence control
ppofolio
PPO ensemble for multi-asset crypto portfolio allocation (BTC/ETH/SOL) with a pre-registered evaluation protocol, walk-forward backtests, and a paper-trading safety chain
dynamic-pricing-lab
Dynamic pricing simulator: Thompson-sampling demand learning, forward-looking customers, and advertising
spark-search-ranking
Counterfactual learning-to-rank for marketplace search logs in PySpark
rankkit
Ranking evaluation with error bars: NDCG, MRR and MAP with confidence intervals, plus position-bias correction for click logs
calikit
Calibration auditing for probabilistic predictions: reliability diagrams, ECE, Brier decomposition, and temperature scaling
