judge-kappa

Unified LLM evaluation that bridges agent-skills-eval (A/B uplift, assertion lists) and MLflow LLMaJ (datasets, named rubric dimensions) while adding statistical rigor missing from both.

Feature matrix

Capability agent-skills-eval MLflow LLMaJ judge-kappa
Native A/B uplift ✅ ❌ ✅
Assertion-based scoring ✅ ❌ ✅
Named rubric dimensions ❌ ✅ ✅
Multi-judge Panel / Jury ❌ ❌ ✅
Krippendorff's α + 95% CI ❌ ❌ ✅
ICC(2,k) variance decomposition ❌ ❌ ✅
Cohen's κ + P(chance) ❌ ❌ ✅
McNemar significance test for uplift ❌ ❌ ✅
Bootstrap CI for mean uplift ❌ ❌ ✅
ICL judge alignment (calibration examples) ❌ ❌ ✅
IRT-based judge weighting (2PL) ❌ ❌ ✅
Per-judge person-fit (outfit MNSQ) ❌ ❌ ✅
Positional bias detection ❌ ❌ ✅
Verbosity bias detection ❌ ❌ ✅
Differential Item Functioning (DIF) ❌ ❌ ✅
Behavioral alignment metric (DISC-style) ❌ ❌ ✅
Pairwise preference rates ❌ ❌ ✅
N-system Elo tournament (N ≤ 6) ❌ ❌ ✅
Listwise ranking for N ≥ 7 (RankJudge) ❌ ❌ ✅
CLI-first ✅ ❌ ✅

Installation

From PyPI (recommended)

# Core — no LLM providers
pip install judge-kappa
uv add judge-kappa

# With Anthropic support
pip install "judge-kappa[anthropic]"
uv add "judge-kappa[anthropic]"

# With OpenAI-compatible support (OpenAI, vLLM, Ollama, OpenRouter, Groq, Together)
pip install "judge-kappa[openai]"
uv add "judge-kappa[openai]"

# Everything including tiktoken for accurate verbosity-bias token counts
pip install "judge-kappa[all]"
uv add "judge-kappa[all]"

From source

git clone https://github.com/williamcaban/judge-kappa-eval
cd judge-kappa-eval

# Python 3.12+ required
uv sync --all-extras          # recommended
# or: pip install -e '.[all]'

API keys

export ANTHROPIC_API_KEY="sk-ant-..."
export OPENAI_API_KEY="sk-..."
# For local vLLM/Ollama, set base_url in the config instead of an API key

Quick start

# Validate a config
judge-kappa validate examples/config_skill_minimal.yaml

# Run evaluation
judge-kappa run examples/config_skill_minimal.yaml

# JSON output → file
judge-kappa run examples/config_skill.yaml --format json --output report.json

# Print config JSON Schema (for editor autocomplete)
judge-kappa schema

Runnable demos (no API key required)

# 6 psychometric methods: bootstrap CI, ICC, person-fit, DIF, behavioral alignment, McNemar
python examples/demo_psychometric.py

# IRT-based judge weighting from calibration data
python examples/demo_irt_weighting.py

# Listwise ranking for 8 systems (98% fewer judge calls than tournament)
python examples/demo_rank_judge.py

Documentation

Doc Contents
AGENTS.md Start here for agents/harnesses — repo identity, science table, quick-start, contributing rules
Decision guide Which mode to use — decision tree covering input format, judge strategy, bias detection, N systems, psychometric rigor, and providers
Input types All entry points: evaluate_skill, evaluate_dataset, evaluate_endpoints, evaluate_prerecorded, evaluate_pairwise_dataset, TournamentEvaluator, RankJudge
Pairwise, Tournament, and Listwise Pairwise preference, N-system Elo tournament, RankJudge (N ≥ 7), champion-challenger; cost tables; mode selection guide
Examples 12 worked examples: 9 YAML configs + 3 Python demos with expected output
Key management api_key_env three-case rule; provider quick reference; mixed-provider panel
CLI reference run, validate, schema commands; output formats; --verdicts flag; CI/CD integration
Python API 13 usage patterns; all new v0.2 patterns (McNemar, IRT weighting, RankJudge, DIF, behavioral alignment); report field reference
Config reference Every YAML field documented; skill/dataset/pairwise mode schemas
Architecture Package structure; ABCs and extension points; data flow diagram
Psychometric methods In-depth guide to all 9 psychometric capabilities: when to use, API, interpretation tables
Scholarly references 18 techniques × (description + in-library use + primary citations); consolidated bibliography

Example configs and scripts

File API key? What it demonstrates
examples/config_skill_minimal.yaml yes Single judge, fastest start
examples/config_skill.yaml yes 3-judge panel + ICL calibration + positional bias
examples/config_dataset.yaml yes Diverse JudgeJury with weighted rubric dimensions
examples/config_dataset_panel.yaml yes Homogeneous panel — rubric consistency check
examples/config_endpoints.yaml yes Per-variant generation backend (endpoint A vs B)
examples/config_prerecorded.yaml yes Score pre-recorded outputs (no generation calls)
examples/config_pairwise.yaml yes Pairwise preference rates (PairwiseReport)
examples/config_regulatory.yaml yes Full bias evidence for compliance submissions
examples/config_mixed_panel.yaml yes Anthropic + OpenRouter + vLLM + Ollama in one panel
examples/demo_psychometric.py no Bootstrap CI, ICC, person-fit, DIF, behavioral alignment, McNemar
examples/demo_irt_weighting.py no IRT 2PL judge weighting from calibration data
examples/demo_rank_judge.py no Listwise ranking for 8 systems (98% call reduction)

Development

uv sync --group dev
uv run pytest                                  # tests (151 pass, 0 fail)
uv run pytest --cov --cov-report=term-missing  # with coverage
uv run ruff check src tests                    # lint
uv run mypy src                                # type check

License

Apache 2.0 — github.com/williamcaban/judge-kappa-eval

Project links

Data verified by PyPI on Aug 23, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for williamcaban from gravatar.com williamcaban

Credits

Author: William Caban

GitHub Statistics

Data verified by PyPI on Aug 23, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache Software License (Apache License)

Requires

Python >=3.12

Provides Extra

all anthropic openai tiktoken

Tags

agent-skills-eval ai-evaluation bias-detection cohen-kappa elo evaluation inter-rater-agreement judge krippendorff llm llm-evaluation mlflow pairwise positional-bias rubric tournament verbosity-bias

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

judge_kappa-0.2.1.tar.gz (126.7 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

judge_kappa-0.2.1-py3-none-any.whl (64.4 kB view details)

Uploaded Python 3

File details

Details for the file judge_kappa-0.2.1.tar.gz.

File metadata

  • Download URL: judge_kappa-0.2.1.tar.gz
  • Upload date:
  • Size: 126.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for judge_kappa-0.2.1.tar.gz
Algorithm Hash digest
SHA256 f0c888ebbd5025bc2dc30249deb6d626573e9151bcf3d3f9d4fd0615e18394db
MD5 4b2ab3d7ec2179d978373d09fc43ec1a
BLAKE2b-256 736232a84175b0b6dba01fba26a9d8aa485336a95cced89851108e094c129511

See more details on using hashes here.

Provenance

The following attestation bundles were made for judge_kappa-0.2.1.tar.gz:

Publisher: publish.yml on williamcaban/judge-kappa-eval

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file judge_kappa-0.2.1-py3-none-any.whl.

File metadata

  • Download URL: judge_kappa-0.2.1-py3-none-any.whl
  • Upload date:
  • Size: 64.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for judge_kappa-0.2.1-py3-none-any.whl
Algorithm Hash digest
SHA256 d23a296f787788dde456c8788fc1fd77fe4f946baa132793c3387057d157dc31
MD5 88d971152531b73e98444da39db65d68
BLAKE2b-256 ced912ef7b223ba1d026d58ad2f5dcb80b7fd15e0a7e9adcf88242deb482f122

See more details on using hashes here.

Provenance

The following attestation bundles were made for judge_kappa-0.2.1-py3-none-any.whl:

Publisher: publish.yml on williamcaban/judge-kappa-eval

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.2.1 This release

2 files

0.2.0

2 files

0.1.0

2 files

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page