raterkit

Reliability, rogue-rater, drift and leakage diagnostics for labelled data.

Every benchmark score, every fine-tune, every "our judge agrees with humans 92% of the time" bottoms out in a pile of human labels — produced by raters who may have been careless, biased, tired by the end of the queue, or graded against gold that was quietly wrong. None of that is visible in accuracy numbers computed on top. raterkit reads the raw ratings and measures each of these failure modes directly, with a confidence interval on every number and a flag only when the evidence clears a stated bar.

The audit in one table

Eight synthetic annotation projects, seven implanted defects. Bold values are flags; each lands on the row where its defect was implanted, and the clean row carries none.

The row to sit with is easy-but-skewed: raters are 90% accurate and agree with each other 81% of the time, yet alpha is 0.34 — far below the 0.667 floor. With 94% of items in one class, raters agree that often by accident; the skew, not the raters, is doing the agreeing. Raw agreement percentages cannot be compared across datasets with different label distributions, which is why every agreement number here is chance-corrected.

make demo regenerates the table, the full report and the figures from fixed seeds; CI rebuilds them from pinned dependencies and fails if a committed number differs from what the code produces.

211,225 real ratings, audited

examples/goemotions/ audits the raw crowd ratings behind GoEmotions — 211,225 ratings by 82 raters over 58,011 Reddit comments, 28 binary emotion labels each. Committed results: examples/goemotions/results/goemotions.md.

• 27 of 28 emotions sit below the 0.667 reliability floor — the whole 95% CI, not just the point estimate. Only gratitude (alpha 0.72) passes; the median emotion manages 0.23, and neutral itself scores 0.26.
• Rater quality spans 0.11 to 0.36 (rater-vs-rest alpha across the 67 raters with 300+ ratings) — the noisiest raters agree with their co-raters at a third of the rate the best ones do, and 20 of 67 sit below 0.20.
• The rater count was never enough for the task. By Spearman-Brown, the observed reliability would need ~13 raters per item to yield 0.8-reliable aggregated labels; the dataset averages 3.6, which projects to 0.52.
• 86% of multi-rated comments are contested on at least one emotion, yet raters marked only 1.6% of ratings "very unclear" — disagreement without awareness of disagreement.

None of this says GoEmotions is a scandal — fine-grained emotion is genuinely hard, and the released splits aggregate and filter these raw ratings. It says the raw layer under a famous benchmark fails the standard reliability bar almost everywhere, measurably and reproducibly.

Install

pip install raterkit

Python 3.11+. Runtime dependencies: numpy, pydantic, matplotlib.

Quickstart

Log one JSON object per rating — who labeled what, and optionally when, in which split, and the item text:

{"item_id": "c41", "rater_id": "r07", "label": "toxic", "t": 1718040000, "split": "train", "text": "..."}

from raterkit import load_ratings, load_gold, run_audit

audit = run_audit(load_ratings("ratings.jsonl"), gold=load_gold("gold.jsonl"))
for flag in audit.flags:
    print(flag.name, "--", flag.detail)

Or from the shell, with an exit code CI can gate on:

raterkit report ratings.jsonl --gold gold.jsonl --out audit --fail-on-flags

An optional config sets the measurement level (nominal, ordinal, interval), the label order, and the target reliability. Alongside report.md the audit writes report.json for pipelines that gate on specific numbers. Any check that could not run is listed with the exact fields to log to enable it — a skip describes the log file, not the raters.

An LLM judge is just another rater: give its verdicts a rater_id and the same audit reports how it agrees with the humans, whether its label distribution diverges, and whether it drifts — no special casing needed.

What it checks

The last three are descriptive and never flag: they explain the numbers above them.

How the numbers are defended

• Cross-checked implementations. Krippendorff's alpha (nominal, ordinal and interval, with missing data) matches the reference krippendorff package to 1e-10; Cohen's and Fleiss' kappa match statsmodels; the chi-square survival function matches scipy to 1e-10; Benjamini-Hochberg reproduces statsmodels decisions exactly. The library itself depends on none of them.
• Validation by implantation. The synthetic projects have defect dials — a spammer, a leaning rater, a competence drift, a gold error rate — and the tests require each probe to recover its dial, stay monotone in it, and stay silent on the clean project.
• Uncertainty on every number. Band probes trigger only when the whole 95% bootstrap CI leaves the innocent band; the bootstrap resamples items (never single ratings), so dependence within an item is respected.
• Drift-checked results. The demo table, report and README numbers are regenerated by CI from pinned dependencies and diffed against the committed copy.

Design notes

• Chance correction is not optional. The easy-but-skewed demo row and the kappa-paradox literature both show raw agreement rewarding class skew. All agreement numbers are Krippendorff's alpha, which handles missing ratings, any number of raters, and nominal/ordinal/interval labels in one framework — and reduces to the familiar coefficients where they overlap.
• Innocent bands, not point nulls. With enough data any rater's marginal "significantly" differs from the pool's while meaning nothing. Band probes state the interval a healthy project could occupy (bias within 0.10 total variation, drift within ±0.10 alpha) and flag only when the whole CI leaves it. Exact procedures use conventions: the 0.667 floor, zero tolerance for cross-split duplicates. All constants live in one place (probes.py) and are easy to disagree with.
• Raters are compared to the pool, not to a standard. A rater is rogue when their CI sits below the pool's CI — a claim about this project, not about an absolute bar their domain may not share.
• Ambiguity is measured, not punished. Contested items are reported descriptively: persistent disagreement can be signal about the task, not noise from the raters, and deserves a number rather than a flag.
• Deterministic to the digit. Seeded PCG64 everywhere, no BLAS in any statistic's path: the same ratings file gives the same report on any machine.

Limitations

• An audit is bounded by what was logged. No timestamps, no drift check; no text or splits, no leakage check; no gold, no gold check.
• The innocent bands and the 0.667 floor are conventions, not derivations; fields differ, and the constants are one file away.
• Spearman-Brown projections assume added raters behave like existing ones — an optimistic bound, stated as such in the report.
• The rogue-rater comparison needs overlap; raters who never co-rate an item with others cannot be tested and are counted as untested, not innocent.
• Implantation shows the probes detect the mechanisms simulated; real annotation projects can fail in ways not simulated here.
• The GoEmotions audit measures the raw rating layer; the released benchmark splits aggregate and filter it, so per-label alpha here is not a direct statement about scores computed on those splits.

Related work

The failure modes measured here are the standard ones from the agreement and annotation-quality literature: agreement coefficients and their proper use (Artstein & Poesio, 2008; Krippendorff, Content Analysis), interpretation benchmarks (Landis & Koch, 1977), the kappa paradox under skewed prevalence (Feinstein & Cicchetti, 1990), crowd label quality (Snow et al., 2008), learning rater competence (Raykar et al., 2010), annotation error detection (Klie et al., 2023), and human label variation as signal (Plank, 2022). The case study audits the GoEmotions raw ratings (Demszky et al., 2020). raterkit's contribution is packaging the checks as one auditable tool with uncertainty on every number and validation by implantation.

Companion projects: judgekit audits LLM judges the way this audits human raters; judgepanel estimates how accurate a panel of judges is when no gold labels exist at all; abkit applies the same audit-before-you-trust pattern to A/B-test readouts; trajectory-judge measures what outcome-only judges miss on agent trajectories; arenakit audits the pairwise leaderboards those labels and judgements end up building.

Citation

Releases are archived on Zenodo. Cite the concept DOI 10.5281/zenodo.21810143, which always resolves to the latest version; structured metadata is in CITATION.cff.

@software{mohammadi_raterkit,
  author  = {Mohammadi, Hadi},
  title   = {raterkit: audit a labeled dataset before you trust it},
  url     = {https://github.com/mohammadi-hadi/raterkit},
  doi     = {10.5281/zenodo.21810143},
  version = {0.1.0},
  year    = {2026}
}

License

MIT