interrater

Inter-rater agreement for systematic reviews and other studies where not every rater sees every subject.

This release reserves the package name. It is not yet usable. Version 2 is under active development; follow the repository for progress.

Background

interrater is the successor to Inter-Rater, described in arXiv:1809.05731 (2018) and archived at Zenodo. Version 1 has been used in published systematic reviews to compute Fleiss' kappa and pairwise Cohen's kappa across multiple reviewers.

Version 2 is a complete rewrite with a different API. Version 1 remains available at git tag v1.5.

What version 2 adds

• Bootstrap confidence intervals. Version 1 reported an analytic standard error for the average of pairwise kappas, which treats those kappas as independent. They are not: two pairwise coefficients sharing a rater are correlated, and the resulting interval is too narrow. Resampling subjects preserves that correlation without having to model it.

• An auditable cleaning layer. Which strings meant nothing, which were repaired and into what, how thin the data was allowed to get: declared in a versioned config file and reported afterwards, rather than reconstructed from memory when a reviewer asks.

• Simulated ratings with known truth. Generate data from an explicit model of rater accuracy, category prevalence and subject assignment, so an estimator can be checked against an answer known in advance and a confidence interval can be checked for actually covering it.

• Support for unbalanced designs. Where each subject is rated by a subset of the reviewer pool rather than all of them, which is what most reviews can actually afford.

Requirements

Python 3.10 or later.

License

GNU General Public License v3 or later. Same license as version 1.