judge-kappa

Unified LLM evaluation that bridges agent-skills-eval (A/B uplift, assertion lists) and MLflow LLMaJ (datasets, named rubric dimensions) while adding statistical rigor missing from both.

Feature matrix

Installation

From PyPI (recommended)

# Core — no LLM providers
pip install judge-kappa
uv add judge-kappa

# With Anthropic support
pip install "judge-kappa[anthropic]"
uv add "judge-kappa[anthropic]"

# With OpenAI-compatible support (OpenAI, vLLM, Ollama, OpenRouter, Groq, Together)
pip install "judge-kappa[openai]"
uv add "judge-kappa[openai]"

# Everything including tiktoken for accurate verbosity-bias token counts
pip install "judge-kappa[all]"
uv add "judge-kappa[all]"

From source

git clone https://github.com/williamcaban/judge-kappa-eval
cd judge-kappa-eval

# Python 3.12+ required
uv sync --all-extras          # recommended
# or: pip install -e '.[all]'

API keys

export ANTHROPIC_API_KEY="sk-ant-..."
export OPENAI_API_KEY="sk-..."
# For local vLLM/Ollama, set base_url in the config instead of an API key

Quick start

# Validate a config
judge-kappa validate examples/config_skill_minimal.yaml

# Run evaluation
judge-kappa run examples/config_skill_minimal.yaml

# JSON output → file
judge-kappa run examples/config_skill.yaml --format json --output report.json

# Print config JSON Schema (for editor autocomplete)
judge-kappa schema

Runnable demos (no API key required)

# 6 psychometric methods: bootstrap CI, ICC, person-fit, DIF, behavioral alignment, McNemar
python examples/demo_psychometric.py

# IRT-based judge weighting from calibration data
python examples/demo_irt_weighting.py

# Listwise ranking for 8 systems (98% fewer judge calls than tournament)
python examples/demo_rank_judge.py

Documentation

Example configs and scripts

Development

uv sync --group dev
uv run pytest                                  # tests (151 pass, 0 fail)
uv run pytest --cov --cov-report=term-missing  # with coverage
uv run ruff check src tests                    # lint
uv run mypy src                                # type check

License

Apache 2.0 — github.com/williamcaban/judge-kappa-eval