interrater-lm

About

A Python package for evaluating language model-style structured outputs with agreement coefficients and statistical procedures for deriving uncertainty and significance.

The aim of this package is not primarily to provide novel implementations of agreement coefficients but instead to make computing these simpler for comparing language models. To that end, we make use of existing libraries for some definitions, and define our own only when well-maintained implementations do not exist or significant additional preprocessing is needed to use with language model-style structured outputs.

Documentation

The documentation is at https://leesadie.github.io/interrater-lm.

Features

• Agreement coefficients
  • Cohen's kappa - as defined by scikit-learn
  • Fleiss's kappa - as defined by statsmodels
  • Gwet's AC1 and AC2
  • Krippendorff's alpha - as defined by fast-krippendorff
  • Kendall's W - derived from scipy's Friedman test
  • Scott's pi
  • Percent agreement
  • Jaccard similarity
  • MASI similarity
• Data levels of measurement
  • Nominal
  • Ordinal
  • Ratio
  • Interval
• Weighting - for Cohen's kappa and Gwet's AC2
  • Identity
  • Linear
  • Quadratic
• Statistical procedures
  • Bias-corrected and accelerated (BCa) cluster bootstrapped confidence intervals - as defined by scipy
  • Permutation tests with optional Monte Carlo sampling for one-sided and two-sided hypotheses
• Standardized data container
• Automatic preprocessing and validation based on data level

Installation

GitHub

• Main branch: https://github.com/leesadie/interrater
• Tags: https://github.com/leesadie/interrater/tags

PyPi

To obtain the latest release of interrater-lm using pip:

python -m pip install interrater_lm

Contributing

Contributions in any form are welcome, including:

• Documentation improvements
• Additional tests
• New features to existing coefficients, e.g., other weights
• New coefficients
• New statistical methods

See Contributing for guidelines.

License

MIT License