Skip to main content

Fairness Indicators

Project description

# Fairness Indicators BETA

![Fairness_Indicators](./images/fairnessIndicators.png)

Fairness Indicators is designed to support teams in evaluating and improving models for fairness concerns in partnership with the broader Tensorflow toolkit.

The tool is currently actively used internally by many of our products, and is now available in BETA for you to try for your own use cases. We would love to partner with you to understand where Fairness Indicators is most useful, and where added functionality would be valuable. Please reach out at tfx@tensorflow.org. You can provide any feedback on your experience, and feature requests, here.

## What is Fairness Indicators? Fairness Indicators enables easy computation of commonly-identified fairness metrics for binary and multiclass classifiers.

Many existing tools for evaluating fairness concerns don’t work well on large scale datasets and models. At Google, it is important for us to have tools that can work on billion-user systems. Fairness Indicators will allow you to evaluate across any size of use case.

In particular, Fairness Indicators includes the ability to:

  • Evaluate the distribution of datasets
  • Evaluate model performance, sliced across defined groups of users * Feel confident about your results with confidence intervals and evals at multiple thresholds
  • Dive deep into individual slices to explore root causes and opportunities for improvement

This [Introductory Video](https://www.youtube.com/watch?v=pHT-ImFXPQo) provides an example of how Fairness Indicators can be used on of our own products to evaluate fairness concerns overtime. This Demo Colab provides a hands-on experience of using Fairness Indicators.

[![](http://img.youtube.com/vi/pHT-ImFXPQo/0.jpg)](http://www.youtube.com/watch?v=pHT-ImFXPQo “”)

The pip package download includes:

  • Tensorflow Data Analysis (TFDV) [analyze distribution of your dataset]
  • Tensorflow Model Analysis (TFMA) [analyze model performance] * Fairness Indicators [an addition to TFMA that adds fairness metrics and the ability to easily compare performance across slices]
  • The What-If Tool (WIT) [an interactive visual interface designed to probe your models better]

## How can I use Fairness Indicators? Tensorflow Models

Not using existing Tensorflow tools? No worries!

Non-Tensorflow Models

## Examples

The [examples](https://github.com/tensorflow/fairness-indicators/tree/master/examples) directory contains several examples.

## More questions? For more information on how to think about fairness evaluation in the context of your use case, see [this link](https://github.com/tensorflow/fairness-indicators/blob/master/documentation/guidance.md).

If you have found a bug in Fairness Indicators, please file a GitHub issue with as much supporting information as you can provide.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for fairness-indicators, version 0.1.0.dev3
Filename, size File type Python version Upload date Hashes
Filename, size fairness_indicators-0.1.0.dev3-py2-none-any.whl (127.6 kB) File type Wheel Python version py2 Upload date Hashes View
Filename, size fairness_indicators-0.1.0.dev3-py3-none-any.whl (127.6 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size fairness_indicators-0.1.0.dev3.tar.gz (121.3 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page