peer-measure 0.0.1

Implementation of the measure Probability of Equal Expected Rank

Probably of Equal Expected Rank

This package is the Python implementation of the MLIR fairness measure "Probability of Equal Expected Rank" using ir_measures.

How to use it

You can either directly install it from PyPi through

pip install peer_measure

Or install the GitHub version

pip install pip@git+https://github.com/hltcoe/peer_measure

When importing, please import both peer_measure and ir_measures.

from peer_measure import PEER
import ir_measures

Please refer to the documentation of ir_measures for the general usage.

Parameters

PEER takes two required parameters: weights and lang_mapping.

• weights: a int-to-float dictionary specifying the weight for each relevance level. The weight have be sum up to 1.0.
• lang_mapping: a str-to-str dictionary with keys being the doc_id and values being the language id of the correspoding document.

You can specify these parameters and the rank cutoff when declaring the measure instance. For example,

measure = PEER(weights={0: 0, 1: 0.5, 2:0, 3: 0.5}, lang_mapping=...)@20

Please refer to our paper for detail definition and implication of the parameters.

Citation

Please consider citing our paper if you use this measure.

@inproceedings{peer,
	author = {Eugene Yang and Thomas Jänich and James Mayfield and Dawn Lawrie},
	title = {Language Fairness in Multilingual Information Retrieval},
	booktitle = {Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (Short Paper) (Accepted)},
	year = {2024}, 
    doi = {10.1145/3626772.3657943}
}