FairRankTune 0.0.0

FairRankTune a python pacakage for fairness-aware data generation, metrics, and algorithms.

FairRankTune

Repository corresponding to the paper.

Updated 8/2023 with additional metrics (score-based group and individual fairness).

Note, the Anonymous 4 open science program does NOT carry over the links below in all instances. To view the file please click on it in the navigation pane. Additionally, if the .ipynb file does not render via the Anonymous 4 opens science UI, please click the "download file" button in the top right to see the file.

Table of Contents

1. Basic installation instructions
2. Quick start & notebook playground
3. FairRankTune source
4. Reproducing experiments
5. Plotting experimental results

Basic Installation Instructions

1. Install via pip (note this is not available due to anonymized review. Once we can we will make fairranktune a pypi package).

pip install FairRankTune

Or install directly from source (right now please download from the anonymous.4open.science repository).

git clone <repo>

2. To use the generator you need to import the function from the package:

import FairRankTune as rt

Quick Start Playground

To use FairRankTune in your run:

import FairRankTune as frt
import numpy as np #helpful to use

To generate a ranking with only the group distribution, number of items, and phi

num_items = 1000
phi = 1  # maximally unfair
group_proportions = np.array([0.2, .3, .5])  # groups of size 200, 300, and 500
ranking, group_ids_in_ranking = frt.GenFromGroups(group_proportions, num_items, phi, 1)

To help get a feel for using FairRankTune we have created a self-contained notebook that includes the FairRankTune source code and metrics. To interact with FairRankTune please see FairRankTunePlayground.ipynb.

FairRankTune Source

All source code for FairRankTune can be found in the FAIRRANKTUNE directory. Each of the three components of the toolkit have their own sub-package/directory described below.

RankTune

The RankTune source code is the RankTune directory and correpsonding src.py script.

Metrics

All metrics are in the Metrics directory. Each metric has a corresponding python script.

• ARP [1]
• AWRF [2]
• ER [3, 4]
• EXPU [4]
• EXPRU [4]
• NDKL [5]
• ERBE [8]
• ERBP [8]
• ERBR [8]
• IAA [9]

Functions to combine per-group metrics into a single metric are in the ComboUtil.py script.

Rankers

All fair ranking methods are in the Rankers directory. Each method has a corresponding python script.

• DetConSort [5]
• Epsilon-Greedy [6]

Functions to combine per-group metrics into a single metric are in the ComboUtil.py script.

Reproducing Demonstration, Case Study, and Comparisons

The scripts to reproduce empirical analysis in the paper can be found in the Analysis directory along with the raw csv files containing the results.

Law Demonstration: the law directory contains the law student data, scripts to run the experiment, result csv, and R script to generate the plots from the paper.

Comparison of RankTune to available methods: the comparison directory contains scripts for the compared methods (including the binary ranked data generation method URG [7], and the result csvs of running the data generation methods.

Case Study: the case_study directory contains scripts and results from generating data and analyzing metrics on the nine distributions from the paper

Timing: the timing directory contains the scripts and results from analyszing the runtime of RankTune.

Generating plots from the paper

The scripts to generate the plots from the case study and comparison with URG are included in the plotting.R file. This script saves off figures to the 'plots' folder in the main directory.

References

[1] Cachel, K., Rundensteiner, E., & Harrison, L. (2022, May). Mani-rank: Multiple attribute and intersectional group fairness for consensus ranking. In 2022 IEEE 38th International Conference on Data Engineering (ICDE) (pp. 1124-1137). IEEE.

@article{Cachel2022MANIRankMA,
  title={MANI-Rank: Multiple Attribute and Intersectional Group Fairness for Consensus Ranking},
  author={Kathleen Cachel and Elke A. Rundensteiner and Lane Harrison},
  journal={2022 IEEE 38th International Conference on Data Engineering (ICDE)},
  year={2022},
  pages={1124-1137}
}

[2] Sapiezynski, P., Zeng, W., E Robertson, R., Mislove, A., & Wilson, C. (2019, May). Quantifying the impact of user attentionon fair group representation in ranked lists. In Companion proceedings of the 2019 world wide web conference (pp. 553-562).

@article{Sapiezynski2019QuantifyingTI,
  title={Quantifying the Impact of User Attentionon Fair Group Representation in Ranked Lists},
  author={Piotr Sapiezynski and Wesley Zeng and Ronald E. Robertson and Alan Mislove and Christo Wilson},
  journal={Companion Proceedings of The 2019 World Wide Web Conference},
  year={2019}
}

[3] Diaz, F., Mitra, B., Ekstrand, M. D., Biega, A. J., & Carterette, B. (2020, October). Evaluating stochastic rankings with expected exposure. In Proceedings of the 29th ACM international conference on information & knowledge management (pp. 275-284).

@article{Diaz2020EvaluatingSR,
  title={Evaluating Stochastic Rankings with Expected Exposure},
  author={Fernando Diaz and Bhaskar Mitra and Michael D. Ekstrand and Asia J. Biega and Ben Carterette},
  journal={Proceedings of the 29th ACM International Conference on Information \& Knowledge Management},
  year={2020}
}

[4] Singh, A., & Joachims, T. (2018, July). Fairness of exposure in rankings. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 2219-2228).

@article{Singh2018FairnessOE,
  title={Fairness of Exposure in Rankings},
  author={Ashudeep Singh and Thorsten Joachims},
  journal={Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery \& Data Mining},
  year={2018}
}

[5] Geyik, S. C., Ambler, S., & Kenthapadi, K. (2019, July). Fairness-aware ranking in search & recommendation systems with application to linkedin talent search. In Proceedings of the 25th acm sigkdd international conference on knowledge discovery & data mining (pp. 2221-2231).

@article{Geyik2019FairnessAwareRI,
  title={Fairness-Aware Ranking in Search \& Recommendation Systems with Application to LinkedIn Talent Search},
  author={Sahin Cem Geyik and Stuart Ambler and Krishnaram Kenthapadi},
  journal={Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery \& Data Mining},
  year={2019}
}

[6] Feng, Y., & Shah, C. (2022). Has CEO Gender Bias Really Been Fixed? Adversarial Attacking and Improving Gender Fairness in Image Search. AAAI Conference on Artificial Intelligence.

@inproceedings{Feng2022HasCG,
  title={Has CEO Gender Bias Really Been Fixed? Adversarial Attacking and Improving Gender Fairness in Image Search},
  author={Yunhe Feng and C. Shah},
  booktitle={AAAI Conference on Artificial Intelligence},
  year={2022}
}

[7] Yang, K., & Stoyanovich, J. (2017, June). Measuring fairness in ranked outputs. In Proceedings of the 29th international conference on scientific and statistical database management (pp. 1-6).

@article{Yang2016MeasuringFI,
  title={Measuring Fairness in Ranked Outputs},
  author={Ke Yang and Julia Stoyanovich},
  journal={Proceedings of the 29th International Conference on Scientific and Statistical Database Management},
  year={2016}
}

[8] Kirnap, Ö., Diaz, F., Biega, A.J., Ekstrand, M.D., Carterette, B., & Yilmaz, E. (2021). Estimation of Fair Ranking Metrics with Incomplete Judgments. Proceedings of the Web Conference 2021.

@article{Kirnap2021EstimationOF,
  title={Estimation of Fair Ranking Metrics with Incomplete Judgments},
  author={{\"O}mer Kirnap and Fernando Diaz and Asia J. Biega and Michael D. Ekstrand and Ben Carterette and Emine Yilmaz},
  journal={Proceedings of the Web Conference 2021},
  year={2021}
}

[9] Biega, A.J., Gummadi, K.P., & Weikum, G. (2018). Equity of Attention: Amortizing Individual Fairness in Rankings. The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval.

@article{Biega2018EquityOA,
  title={Equity of Attention: Amortizing Individual Fairness in Rankings},
  author={Asia J. Biega and Krishna P. Gummadi and Gerhard Weikum},
  journal={The 41st International ACM SIGIR Conference on Research \& Development in Information Retrieval},
  year={2018}
}