Equinox is a powerful and scalable Python library for calculating inter-annotator agreement scores for a variety of Natural Language Processing and Machine Learning tasks.
Project description
EQUINOX
Equinox is a powerful and scalable Python library for calculating inter-annotator agreement scores for a variety of Natural Language Processing and Machine Learning tasks.
For example:
- Fleiss's Kappa: A statistical measure for assessing the reliability of agreement between a fixed number of raters when assigning categorical ratings to a number of items or classifying items.
Many advanced ML tasks require annotating enormous amounts of data, which is a tedious and time-consuming task. Add on the time required to wrangle the data and compute inter-annotator agreement scores, and poof, weeks have gone by. We aim to answer the call for quicker annotation metrics Whether you are an academic, ML Engineer, or Citizen Data Scientist, Equinos is straightforward to install and our goal is to allow you to get to your NLP insights faster.
More funcionality is currently being built, and will be deployed upon completion. Stay tuned!
Installation
Equinox can easily be installed via pip
:
pip install equinox
How to Use Equinox
Equinox is very simple to use. See our documentation for more information. Below is an example to compute Fleiss's Kappa for > 2 annotators.
#Import
import equinox
#Define Input
entities = ["yes", "no"]
num_annotators = 10
df = pd.read_csv("annotated_data.csv") #See below
k = equinox.fleiss_kappa(entities, num_annotators, df)
## 0.53
Where:
entities
A list of the names of entities annotated. These entity names must match the column names indf
.num_annotators
: The number of annotators in the dataset.df
: Pandas DataFrame consiting of the number of annotations per entity (columns) for a finite number of documents (rows):
Interpretting Fleiss's Kappa Coefficients
Dependencies
Equinox requires the following:
Running Tests
Tests are written in the tests
directory and processed using PyTest <https://docs.pytest.org/en/latest/>
__.
Python Version
Equinox supports Python 3.6+ <https://python3statement.org/>
__ and, due to the use of unicode variable names/identifiers, is not compatible with Python 2.x. Given the small dependencies, Equinox may work on older versions of Python but this is beyond the scope of our support and we strongly recommend that you upgrade to the most recent version of Python.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for equinox_gould29-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 552f8485845f5d328f365490d91a2586475ff5dbafc44ce3a52702cb6901d7f8 |
|
MD5 | 77899c79450be0663719b9001bcd1aa1 |
|
BLAKE2b-256 | 4bec1fc647f38733f0e36ad2be4153ec86cec0dd0a3665a483abe207b226d7f2 |