iaa-od: a comprehensive IAA analysis suite

This package provides a comprehensive Inter-Annotator Agreement analysis suite for Object Detection tasks.

Installation

PyPI

iaa-od is available on PyPI, so you can install it through pip:

pip install iaa-od

Source

You can install the iaa-od package by cloning this repository and then running the following command on its top level:

pip install .

If you want to make modifications to the package, you may install it like so:

pip install -e .

If you wish to do some testing through Jupyter Notebooks, install the optional dependencies as well:

pip install -e ".[dev]"

Computed metrics

Below is a list of currently implemented metrics.

Krippendorff's $\alpha$

Krippendorff's $\alpha$ coefficient is defined as:

$$\alpha = 1 - \frac{D_o}{D_e}$$

and is computed in our package according to this paper by Klaus Krippendorff.

The units in each image are found through agglomerative clustering.

Rater vitality

Rater vitality measures the contribution of each annotator to the global agreement value. It is computed as:

$$\nu_a = \alpha - \alpha_a$$

where $\alpha_a$ is the value of Krippendorff's $\alpha$ computed excluding annotator $a$ from the dataset.

Scale complexity

Scale complexity measures the ambiguity found across categories of the same dataset by grouping them into user-defined macro-categories. It is computed as:

$$\text{SC} = \alpha_M - \alpha$$

where $\alpha_M$ is the value of Krippendorff's $\alpha$ computed on the macro-categories. If $\text{SC} > 0$, categories grouped in the same macro-categories are ambiguous.

L-Score (Localisation Score)

L-Score measures the localisation agreement reached by the annotators by only considering the locations of their bounding boxes. It is computed as:

$$L_{\text{score}} = \frac{|A|}{2|A|-1} \left[ \sum_{i=2}^{|A|} \left(\frac{i-1}{|A|}, p_i\right) - p_1 + 1\right]$$

L-Score $\in [0,1]$, where 1 indicates perfect localisation agreement.

C-Score (Categorisation Score)

C-Score measures the categorisation agreement reached by annotators by analysing intersecting bounding boxes and comparing their labels.

Computation is not reported here for brevity.

C-Score $\in [0,1]$, where 1 indicates perfect categorisation agreement.

Count and size granularity

Count and size granularity measure the behaviour of annotators when annotating large groups of object:

• an annotator who groups many instances into a single bounding box annotates "by group";
• an annotator who creates several bounding boxes for each instance annotates "by instance".

Computation of these metrics is not reported here for brevity.

Count and size granularities both rely on a benchmark annotator, found during computation, to whom all other annotators are compared to.

Working example

main.py provides a working example of the package by performing all available analyses on a dataset with three annotators. More examples are also provided in the experiments directory: specifically, experiments/images.py shows how to utilise the visualisation module.

Jupyter notebooks

The notebooks directory contains a number of Jupyter notebooks to play around with data.