judge-kappa

Unified LLM evaluation that bridges agent-skills-eval (A/B uplift, assertion lists) and MLflow LLMaJ (datasets, named rubric dimensions) while adding statistical rigor missing from both.

Feature matrix

Capability agent-skills-eval MLflow LLMaJ judge-kappa
Native A/B uplift ✅ ❌ ✅
Assertion-based scoring ✅ ❌ ✅
Named rubric dimensions ❌ ✅ ✅
Multi-judge Panel / Jury ❌ ❌ ✅
Krippendorff's α + 95% CI ❌ ❌ ✅
ICC(2,k) variance decomposition ❌ ❌ ✅
Cohen's κ + P(chance) ❌ ❌ ✅
McNemar significance test for uplift ❌ ❌ ✅
Bootstrap CI for mean uplift ❌ ❌ ✅
ICL judge alignment (calibration examples) ❌ ❌ ✅
IRT-based judge weighting (2PL) ❌ ❌ ✅
Per-judge person-fit (outfit MNSQ) ❌ ❌ ✅
Positional bias detection ❌ ❌ ✅
Verbosity bias detection ❌ ❌ ✅
Differential Item Functioning (DIF) ❌ ❌ ✅
Behavioral alignment metric (DISC-style) ❌ ❌ ✅
Pairwise preference rates ❌ ❌ ✅
N-system Elo tournament (N ≤ 6) ❌ ❌ ✅
Listwise ranking for N ≥ 7 (RankJudge) ❌ ❌ ✅
CLI-first ✅ ❌ ✅

Installation

From PyPI (recommended)

# Core — no LLM providers
pip install judge-kappa
uv add judge-kappa

# With Anthropic support
pip install "judge-kappa[anthropic]"
uv add "judge-kappa[anthropic]"

# With OpenAI-compatible support (OpenAI, vLLM, Ollama, OpenRouter, Groq, Together)
pip install "judge-kappa[openai]"
uv add "judge-kappa[openai]"

# Everything including tiktoken for accurate verbosity-bias token counts
pip install "judge-kappa[all]"
uv add "judge-kappa[all]"

From source

git clone https://github.com/williamcaban/judge-kappa-eval
cd judge-kappa-eval

# Python 3.12+ required
uv sync --all-extras          # recommended
# or: pip install -e '.[all]'

API keys

export ANTHROPIC_API_KEY="sk-ant-..."
export OPENAI_API_KEY="sk-..."
# For local vLLM/Ollama, set base_url in the config instead of an API key

Quick start

# Validate a config
judge-kappa validate examples/config_skill_minimal.yaml

# Run evaluation
judge-kappa run examples/config_skill_minimal.yaml

# JSON output → file
judge-kappa run examples/config_skill.yaml --format json --output report.json

# Print config JSON Schema (for editor autocomplete)
judge-kappa schema

Runnable demos (no API key required)

# 6 psychometric methods: bootstrap CI, ICC, person-fit, DIF, behavioral alignment, McNemar
python examples/demo_psychometric.py

# IRT-based judge weighting from calibration data
python examples/demo_irt_weighting.py

# Listwise ranking for 8 systems (98% fewer judge calls than tournament)
python examples/demo_rank_judge.py

Documentation

Doc Contents
Decision guide Which mode to use — decision tree covering input format, judge strategy, bias detection, N systems, psychometric rigor, and providers
Input types All entry points: evaluate_skill, evaluate_dataset, evaluate_endpoints, evaluate_prerecorded, evaluate_pairwise_dataset, TournamentEvaluator, RankJudge
Pairwise, Tournament, and Listwise Pairwise preference, N-system Elo tournament, RankJudge (N ≥ 7), champion-challenger; cost tables; mode selection guide
Examples 12 worked examples: 9 YAML configs + 3 Python demos with expected output
Key management api_key_env three-case rule; provider quick reference; mixed-provider panel
CLI reference run, validate, schema commands; output formats; --verdicts flag; CI/CD integration
Python API 13 usage patterns; all new v0.2 patterns (McNemar, IRT weighting, RankJudge, DIF, behavioral alignment); report field reference
Config reference Every YAML field documented; skill/dataset/pairwise mode schemas
Architecture Package structure; ABCs and extension points; data flow diagram
Psychometric methods In-depth guide to all 9 psychometric capabilities: when to use, API, interpretation tables
Scholarly references 18 techniques × (description + in-library use + primary citations); consolidated bibliography

Example configs and scripts

File API key? What it demonstrates
examples/config_skill_minimal.yaml yes Single judge, fastest start
examples/config_skill.yaml yes 3-judge panel + ICL calibration + positional bias
examples/config_dataset.yaml yes Diverse JudgeJury with weighted rubric dimensions
examples/config_dataset_panel.yaml yes Homogeneous panel — rubric consistency check
examples/config_endpoints.yaml yes Per-variant generation backend (endpoint A vs B)
examples/config_prerecorded.yaml yes Score pre-recorded outputs (no generation calls)
examples/config_pairwise.yaml yes Pairwise preference rates (PairwiseReport)
examples/config_regulatory.yaml yes Full bias evidence for compliance submissions
examples/config_mixed_panel.yaml yes Anthropic + OpenRouter + vLLM + Ollama in one panel
examples/demo_psychometric.py no Bootstrap CI, ICC, person-fit, DIF, behavioral alignment, McNemar
examples/demo_irt_weighting.py no IRT 2PL judge weighting from calibration data
examples/demo_rank_judge.py no Listwise ranking for 8 systems (98% call reduction)

Development

uv sync --group dev
uv run pytest                                  # tests (151 pass, 0 fail)
uv run pytest --cov --cov-report=term-missing  # with coverage
uv run ruff check src tests                    # lint
uv run mypy src                                # type check

License

Apache 2.0 — github.com/williamcaban/judge-kappa-eval

Project links

Data verified by PyPI on Aug 23, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.2.1)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for williamcaban from gravatar.com williamcaban

Credits

Author: William Caban

GitHub Statistics

Data verified by PyPI on Aug 23, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache Software License (Apache License)

Requires

Python >=3.12

Provides Extra

all anthropic openai tiktoken

Tags

agent-skills-eval ai-evaluation bias-detection cohen-kappa elo evaluation inter-rater-agreement judge krippendorff llm llm-evaluation mlflow pairwise positional-bias rubric tournament verbosity-bias

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

judge_kappa-0.2.0.tar.gz (120.0 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

judge_kappa-0.2.0-py3-none-any.whl (64.3 kB view details)

Uploaded Python 3

File details

Details for the file judge_kappa-0.2.0.tar.gz.

File metadata

  • Download URL: judge_kappa-0.2.0.tar.gz
  • Upload date:
  • Size: 120.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for judge_kappa-0.2.0.tar.gz
Algorithm Hash digest
SHA256 62617385a5bed86005d8418f6ab36dccb082b49971a0c9f64e870fe75a770466
MD5 58726b0cab15d700fc8c7fbb12738b5b
BLAKE2b-256 018b4cf343a6999e91f02a0626fb8c89fd154458f5a33c7d5731a27b117b2ca9

See more details on using hashes here.

Provenance

The following attestation bundles were made for judge_kappa-0.2.0.tar.gz:

Publisher: publish.yml on williamcaban/judge-kappa-eval

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file judge_kappa-0.2.0-py3-none-any.whl.

File metadata

  • Download URL: judge_kappa-0.2.0-py3-none-any.whl
  • Upload date:
  • Size: 64.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for judge_kappa-0.2.0-py3-none-any.whl
Algorithm Hash digest
SHA256 365b76300eb8d188c1aee2f104e50d161bec064cb464875e41fcb2a1184d99df
MD5 42f64cd7db1f66afca070d120c9dee89
BLAKE2b-256 b88b36fcdef91c4034a928614db99e48af0c4b0df0750fb2b4e08a412e6456bf

See more details on using hashes here.

Provenance

The following attestation bundles were made for judge_kappa-0.2.0-py3-none-any.whl:

Publisher: publish.yml on williamcaban/judge-kappa-eval

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.2.1

2 files

This release

0.2.0 This release

2 files

0.1.0

2 files

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page