Annotation Analysis
Project description
BRR
Annotation Analysis
Package to analyse inter-annotator agreement.
Simple package to compute Interannotation Agreement with a simple interface.
Getting Started
Install package:
pip3 install annotation_analysis
Example usage:
from annotation_analysis import interannotator_metrics
if __name__ == "__main__":
annotations = [["A","B","A"],["A","B","B"]]
k_alpha = interannotator_metrics.krippendorff_alpha(annotations)
f_kappa = interannotator_metrics.fleiss_kappa(annotations)
print(k_alpha)
print(f_kappa)
Documentation
function krippendorf_alpha(annotations:List[List],labels:Optional[List], ignore=Optional[List])
"""
Arguments:
- annotations:
- The list of annotations, assumed to be one "row" per annotator (i.e. annotations[0] is annotator #1).
- Number of Columns represents the number of datapoints.
- Important: at this stage only hashable values are allowed.
- E.g. with labels ["A","B"], and three datapoints and 2 annotators the following would be the valid structure:
- [["A","B","A"],["A","B","B"]],
- Annotator #1 (index=0) -> ["A","B","A"]
- Annotator #2 (index=1) -> ["A","B","B"]
- labels:
- Represents the optional list of valid labels.
- Important: at this stage only hashable values are allowed.
- ignore:
- Represents a list of optional labels that should be ignored
- I.e. If this is non-empty then for any datapoint, if any of the annotators has the ignored label, the data point with all annotators is ignored
Return:
- Krippendorff Alpha score for all annotators.
"""
function fleiss_kappa(annotations:List[List],labels:Optional[List], ignore=Optional[List])
"""
Arguments:
- annotations:
- The list of annotations, assumed to be one "row" per annotator (i.e. annotations[0] is annotator #1).
- Number of Columns represents the number of datapoints.
- Important: at this stage only hashable values are allowed.
- E.g. with labels ["A","B"], and three datapoints and 2 annotators the following would be the valid structure:
- [["A","B","A"],["A","B","B"]],
- Annotator #1 (index=0) -> ["A","B","A"]
- Annotator #2 (index=1) -> ["A","B","B"]
- labels:
- Represents the optional list of valid labels.
- Important: at this stage only hashable values are allowed.
- ignore:
- Represents a list of optional labels that should be ignored
- I.e. If this is non-empty then for any datapoint, if any of the annotators has the ignored label, the data point with all annotators is ignored
Return:
- Fleiss Kappa score for all annotators.
"""
(C) - Nikolai Rozanov 2022
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for annotation_analysis-0.0.0.3.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | bf9e816257227501e4188f2f816f593dc9d290a39b6aed4a7dcef1dcca56d923 |
|
MD5 | 04297dce5af9c6d229681b256278a5ac |
|
BLAKE2b-256 | 65c4e6f55e25caa5651e5d03afa391a36e9f7ff241d0ac21e210077625281c19 |
Close
Hashes for annotation_analysis-0.0.0.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b1ffdb6b9c318a024449c2af5599a11ca3d60523b05d29ac1f5bb2c8e6981e20 |
|
MD5 | 2e0ec6e5e65bdd25d3227186d81254b6 |
|
BLAKE2b-256 | 5d1b5a6ff7cf740969a80c2df0b8850ea43d26f6123f52d7697cdfe3cf121171 |