Skip to main content

FairMetrics

This repository is the official implementation of the paper "Rethinking Fairness Metrics for Multi-Class and Multi-Group Assessment: Not Everything is Binary."

Authors:

¹ Department of Environmental Sciences, Informatics and Statistics, Ca’ Foscari University of Venice, Via Torino 155, Venezia Mestre, 30172, Italy
² Tilburg Institute for Law, Technology, and Society, Tilburg University, Prof. Cobbenhagenlaan 221, Tilburg, 5037 DB, The Netherlands
³ European Center for Living Technologies (ECLT), Venice, Italy

*Corresponding author
†Equal contribution

Layout

fairmetrics/
├── __init__.py
├── _utils.py
├── metrics.py
├── tests.py
├── independence.py
├── separation.py
├── sufficiency.py
└── accuracy.py

fairmetrics_experiments/
├── __init__.py
├── binarization.py
└── run_hpo.py

scripts/
└── multi_run.sh

The split is organized around:

  • fairmetrics/metrics.py: standard fairness metrics
  • fairmetrics/tests.py: notebook-style data/HPO/test helpers
  • fairmetrics_experiments/run_hpo.py: paper-style HPO runner (adult/students, mul/bin*) + fairness audit

Installation

pip install -e .

Optional experiment dependencies:

pip install -e '.[experiments]'

Modes used in paper runs

The experiment runner supports four target strategies:

  • mul

    • keep the original multiclass target as-is
  • bin0

    • one-vs-rest binarization using class index 0 as positive
    • mapping: y == class_0 -> 1, else 0
  • bin1

    • one-vs-rest binarization using class index 1 as positive
    • mapping: y == class_1 -> 1, else 0
  • bin2

    • one-vs-rest binarization using class index 2 as positive
    • mapping: y == class_2 -> 1, else 0

In short:

  • mul = multiclass experiment
  • bin* = binary one-vs-rest experiment for the selected class index

Fairness configuration used in runs

The runner evaluates fairness with intersectional sensitive attributes and conditional strata:

  • Adult

    • sensitive: ['Sex', 'Race_White']
    • strata: ['Country_United-States']
  • Students

    • sensitive: ['Gender', 'International']
    • strata: ['Displaced']

The output JSON includes:

  • HPO result (best_params, best_score)
  • fairness metrics:
    • demographic_parity
    • conditional_demographic_parity
    • equal_opportunity
    • predictive_equality
    • equalized_odds
    • predictive_parity
    • equal_accuracy
  • intersectional_sensitive_groups

Single-run commands (paper style)

python -m fairmetrics_experiments.run_hpo --dataset adult --mode mul
python -m fairmetrics_experiments.run_hpo --dataset adult --mode bin0
python -m fairmetrics_experiments.run_hpo --dataset adult --mode bin1
python -m fairmetrics_experiments.run_hpo --dataset adult --mode bin2

python -m fairmetrics_experiments.run_hpo --dataset students --mode mul
python -m fairmetrics_experiments.run_hpo --dataset students --mode bin0
python -m fairmetrics_experiments.run_hpo --dataset students --mode bin1
python -m fairmetrics_experiments.run_hpo --dataset students --mode bin2

Save JSON output:

python -m fairmetrics_experiments.run_hpo --dataset adult --mode bin0 --output results/adult_bin0.json

Batch execution

Run the full set of strategies:

bash scripts/multi_run.sh

It runs:

  • Adult: mul, bin0, bin1, bin2
  • Students: mul, bin0, bin1, bin2

and stores outputs in results/, e.g.:

  • results/adult_mul.json
  • results/adult_bin0.json
  • results/adult_bin1.json
  • results/adult_bin2.json
  • results/students_mul.json
  • results/students_bin0.json
  • results/students_bin1.json
  • results/students_bin2.json

FRACTALS Research Group

FRACTALS
Fair and Responsible Algorithms for Complex-systems, Therapeutics, Astrophysics and Life-Sciences

🔗 https://fractals.group

Metadata

Release files for FairMetrics 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for FairMetrics 0.1.0
File Size Uploaded Provenance Actions
fairmetrics-0.1.0.tar.gz 723.0 kB Without provenance Details for fairmetrics-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for FairMetrics 0.1.0
File Interpreter ABI Platform Actions
fairmetrics-0.1.0-py3-none-any.whl Python 3 none any Details for fairmetrics-0.1.0-py3-none-any.whl

Total release size: 1.5 MB

Release files / fairmetrics-0.1.0.tar.gz

Download URL fairmetrics-0.1.0.tar.gz
Size 723.0 kB
Tags Source
SHA-256 checksum
How to use checksums
1634fb61e9d64ff23a1e96f1ddad3523dad0d7d2f65703db0b69df82428888d1
BLAKE2b-256 checksum
How to use checksums
db9299deb4bc79d0054898f5ade416dd3eb9ff26deaa49680c89f11bfe8bb978
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.12

Release files / fairmetrics-0.1.0-py3-none-any.whl

Download URL fairmetrics-0.1.0-py3-none-any.whl
Size 777.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
18323081d688657fed8da0e975a21eab7c59db9d04ccdf3c5ce342d2169ce3f3
BLAKE2b-256 checksum
How to use checksums
fc2e8c621b8de491a15218509ab7b1ef03d6c7169649f415c8f06b23f4bfb287
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.12

Security for FairMetrics 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page