Inter-rater agreement Phi, as an alternative to Kripperndorfs alpha, as described in https://github.com/AlessandroChecco/agreement-phi
Project description
Agreement measure Phi
Source code for inter-rater agreement measure Phi. Live demo here: http://agreement-measure.sheffield.ac.uk
Requirements
python 3+, pymc3 3.3+. See requirements files for tested working versions on linux and osx.
Installation - with pip
Simply run pip install agreement_phi
.
This will provide a module and a command line executable called run_phi
.
Installation - without pip
Download the folder.
Example - from command line
Prepare a csv file (no header, each row is a document, each column a rater), leaving empty the missing values. For example input.csv
:
1,2,,3
1,1,2,
4,3,2,1
And execute from the console run_phi -f input.csv
.
Example - from python
Input is a numpy 2-dimensional array with NaN for missing values, or equivalently a python list of lists (where each list is a set of ratings for a document, with arbitrary length). Every row represents a different document, every column a different rating. Note that Phi does not take in account rater bias, so the order in which ratings appear for each document does not matter. For this reasons, missing values and a sparse representation is needed only when documents have different number of ratings.
Input example
import numpy as np
m_random = np.random.randint(5, size=(5, 10)).tolist()
m_random[0][1]=np.nan
or equivalently
m_random = np.random.randint(5, size=(5, 10)).astype(float)
m_random[0][1]=np.nan
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for agreement_phi-0.2.5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 59bdda31a832047b872aa4201e12c7425ebbf454727904288f595c2e5bc73833 |
|
MD5 | 80a224c2a5f8362751449df908e74949 |
|
BLAKE2b-256 | dedeafd3cace366291550bf27b44f59db64a22d0a6bf211982c0e745ba3ee4bd |