ml-agreement 1.0.0

A metric to calculate inter-annotator agreement for scenarios where several labels can be simultaneously assigned to an item

Multi-Label Agreement (ml_agreement)

Multi-Label Agreement (ml_agreement) is a Python library for measuring inter-annotator agreement in multi-label annotation tasks — scenarios where annotators can assign multiple labels (or none) to each data item. Unlike traditional agreement metrics like Cohen's Kappa or Fleiss’ Kappa, this metric is tailored for tasks where each item may receive multiple simultaneous annotations.

Key Features

• Supports multi-label annotations (binary one-hot label vectors)
• Handles any number of annotators and labels
• Compatible with standard agreement normalizations: kappa, AC1, and PABAK
• Calculates agreement:
  • For each item
  • For each label
  • Overall across the dataset
• Provides baseline estimates like expected agreement and minimum possible agreement

Background

Inter-annotator agreement (IAA) is essential to ensure the objectivity and quality of annotated data. However, most classical metrics assume a single-label decision per annotator per item.

In contrast, many real-world NLP and social science tasks involve multi-label annotation, e.g., tagging media bias, emotions, or topics — where a sentence might validly be tagged with multiple classes.

This package defines a new agreement framework adapted for multi-label settings and implements a range of calculations to assess agreement reliability.

For a deeper discussion, see our paper ...

Installation

You can install this package via pip or clone it directly from git:

pip install ml_agreement

https://github.com/Timperator2/MultiLabelAgreement.git

Usage Example

My Project

import ml_agreement

# Each annotator assigns k labels to N items
annotator_1 = [["Science Fiction", "Action", "Comedy"], ["Drama", "Romance"], ["Horror", "Comedy"]]
annotator_2 = [["Science Fiction", "Action"], ["Romance"], ["Horror", "Comedy"]]
annotator_3 = [["Science Fiction", "Action"], ["Drama", "Romance"], ["Thriller", "Comedy"]]

all_decisions = [annotator_1, annotator_2, annotator_3]

# The data needs to be converted to binary one-hot label vectors

# The labels available to assign to each item
available_labels = ["Action", "Comedy", "Drama", "Horror", "Romance", "Science Fiction", "Thriller", "Fantasy"]

# If the data is already binary encoded, this transformation converts it back to the textual form as above
all_decisions = ml_agreement.transform_data(all_decisions, available_labels)

# Calculate the multi-label agreement using all three different methods 
# (for a discussion of each method, see the paper referenced above)
multi_label_k = ml_agreement.calculate_total_multi_label_agreement(all_decisions, "k")
multi_label_ac1 = ml_agreement.calculate_total_multi_label_agreement(all_decisions, "ac1")
multi_label_pabak = ml_agreement.calculate_total_multi_label_agreement(all_decisions, "pabak")

# Calculate the minimum possible multi-label agreement to see how low it can drop 
# with the given number of annotators and prevalence
minimum_k = ml_agreement.calculate_minimum_prevalence_agreement(all_decisions, "k")

# Calculate the prevalence of a label (in this case label_0, which is "Action")
prevalence_action = ml_agreement.calculate_label_prevalence(all_decisions, 0)

# Calculate the agreement for this label
ml_agreement_k_action = ml_agreement.calculate_label_agreement(all_decisions, 0, "k")

License

MIT License. See LICENSE file for details.

Citation

If you use this package in your research, please cite the corresponding paper.