FairMetrics
This repository is the official implementation of the paper "Rethinking Fairness Metrics for Multi-Class and Multi-Group Assessment: Not Everything is Binary."
Authors:
- Matteo Grazioso¹*† (matteo.grazioso@unive.it)
- Fabio Naccarella¹† (900619@stud.unive.it)
- Chiara Gallese² (C.Gallese@tilburguniversity.edu)
- Marco S. Nobile¹ ³ (marco.nobile@unive.it)
¹ Department of Environmental Sciences, Informatics and Statistics, Ca’ Foscari University of Venice, Via Torino 155, Venezia Mestre, 30172, Italy
² Tilburg Institute for Law, Technology, and Society, Tilburg University, Prof. Cobbenhagenlaan 221, Tilburg, 5037 DB, The Netherlands
³ European Center for Living Technologies (ECLT), Venice, Italy
*Corresponding author
†Equal contribution
Layout
fairmetrics/
├── __init__.py
├── _utils.py
├── metrics.py
├── tests.py
├── independence.py
├── separation.py
├── sufficiency.py
└── accuracy.py
fairmetrics_experiments/
├── __init__.py
├── binarization.py
└── run_hpo.py
scripts/
└── multi_run.sh
The split is organized around:
fairmetrics/metrics.py: standard fairness metricsfairmetrics/tests.py: notebook-style data/HPO/test helpersfairmetrics_experiments/run_hpo.py: paper-style HPO runner (adult/students,mul/bin*) + fairness audit
Installation
pip install -e .
Optional experiment dependencies:
pip install -e '.[experiments]'
Modes used in paper runs
The experiment runner supports four target strategies:
-
mul- keep the original multiclass target as-is
-
bin0- one-vs-rest binarization using class index 0 as positive
- mapping:
y == class_0 -> 1, else0
-
bin1- one-vs-rest binarization using class index 1 as positive
- mapping:
y == class_1 -> 1, else0
-
bin2- one-vs-rest binarization using class index 2 as positive
- mapping:
y == class_2 -> 1, else0
In short:
mul= multiclass experimentbin*= binary one-vs-rest experiment for the selected class index
Fairness configuration used in runs
The runner evaluates fairness with intersectional sensitive attributes and conditional strata:
-
Adult
- sensitive:
['Sex', 'Race_White'] - strata:
['Country_United-States']
- sensitive:
-
Students
- sensitive:
['Gender', 'International'] - strata:
['Displaced']
- sensitive:
The output JSON includes:
- HPO result (
best_params,best_score) - fairness metrics:
demographic_parityconditional_demographic_parityequal_opportunitypredictive_equalityequalized_oddspredictive_parityequal_accuracy
intersectional_sensitive_groups
Single-run commands (paper style)
python -m fairmetrics_experiments.run_hpo --dataset adult --mode mul
python -m fairmetrics_experiments.run_hpo --dataset adult --mode bin0
python -m fairmetrics_experiments.run_hpo --dataset adult --mode bin1
python -m fairmetrics_experiments.run_hpo --dataset adult --mode bin2
python -m fairmetrics_experiments.run_hpo --dataset students --mode mul
python -m fairmetrics_experiments.run_hpo --dataset students --mode bin0
python -m fairmetrics_experiments.run_hpo --dataset students --mode bin1
python -m fairmetrics_experiments.run_hpo --dataset students --mode bin2
Save JSON output:
python -m fairmetrics_experiments.run_hpo --dataset adult --mode bin0 --output results/adult_bin0.json
Batch execution
Run the full set of strategies:
bash scripts/multi_run.sh
It runs:
- Adult:
mul,bin0,bin1,bin2 - Students:
mul,bin0,bin1,bin2
and stores outputs in results/, e.g.:
results/adult_mul.jsonresults/adult_bin0.jsonresults/adult_bin1.jsonresults/adult_bin2.jsonresults/students_mul.jsonresults/students_bin0.jsonresults/students_bin1.jsonresults/students_bin2.json
FRACTALS Research Group
FRACTALS
Fair and Responsible Algorithms for Complex-systems, Therapeutics, Astrophysics and Life-Sciences
Metadata
Release files for FairMetrics 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| fairmetrics-0.1.0.tar.gz | 723.0 kB | Without provenance | Details for fairmetrics-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| fairmetrics-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for fairmetrics-0.1.0-py3-none-any.whl |
Total release size: 1.5 MB
Release files / fairmetrics-0.1.0.tar.gz
| Download URL | fairmetrics-0.1.0.tar.gz |
|---|---|
| Size | 723.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1634fb61e9d64ff23a1e96f1ddad3523dad0d7d2f65703db0b69df82428888d1
|
|
BLAKE2b-256 checksum How to use checksums |
db9299deb4bc79d0054898f5ade416dd3eb9ff26deaa49680c89f11bfe8bb978
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.12
|
Release files / fairmetrics-0.1.0-py3-none-any.whl
| Download URL | fairmetrics-0.1.0-py3-none-any.whl |
|---|---|
| Size | 777.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
18323081d688657fed8da0e975a21eab7c59db9d04ccdf3c5ce342d2169ce3f3
|
|
BLAKE2b-256 checksum How to use checksums |
fc2e8c621b8de491a15218509ab7b1ef03d6c7169649f415c8f06b23f4bfb287
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.12
|
Security for FairMetrics 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.