veritastool 0.8.7

veritastool

Veritas Toolkit

The purpose of this toolkit is to facilitate the adoption of Veritas Methodology on Fairness Assessment and spur industry development. It will also benefit customers by improving the fairness of financial services delivered by AIDA systems.

Installation

The easiest way to install veritastool is to download it from PyPI. It's going to install the library itself and its prerequisites as well.

pip install veritastool

Then, you will be able to import the library and use its functionalities. Before we do that, we can run some test functions on our sample datasets to see if our codes are performing as expected.

from veritastool.utility import test_function_cs
test_function_cs()

from veritastool.utility import test_function_cm
test_function_cm()

Output:

Initialization

You can now import the custom library that you would to use for diagnosis. In this example we will use the Credit Scoring custom library.

from veritastool import ModelContainer
from veritastool.fairness import CreditScoring

Once the relevant use case object (CreditScoring) and model container (ModelContainer) has been imported, you can upload your contents into the container and initialize the object for diagnosis.

import pickle
import numpy as np

#Load Credit Scoring Test Data
file = "./veritastool/resources/data/credit_score_dict.pickle"
input_file = open(file, "rb")
cs = pickle.load(input_file)

#Reduce into two classes
cs["X_train"]['MARRIAGE'] = cs["X_train"]['MARRIAGE'].replace([0, 3],1)
cs["X_test"]['MARRIAGE'] = cs["X_test"]['MARRIAGE'].replace([0, 3],1)

#Model Container Parameters
y_true = np.array(cs["y_test"])
y_pred = np.array(cs["y_pred"])
y_train = np.array(cs["y_train"])
p_var = ['SEX', 'MARRIAGE']
p_grp = {'SEX': [1], 'MARRIAGE':[1]}
x_train = cs["X_train"]
x_test = cs["X_test"]
model_object = cs["model"]
model_name = "credit scoring"
model_type = "default"
y_prob = cs["y_prob"]

container = ModelContainer(y_true = y_true, y_train = y_train, p_var = p_var, p_grp = p_grp, x_train = x_train,  x_test = x_test, model_object = model_object, model_type  = model_type,model_name =  model_name, y_pred= y_pred, y_prob= y_prob)

cre_sco_obj= CreditScoring(model_params = [container], fair_threshold = 0.43, fair_concern = "eligible", fair_priority = "benefit", fair_impact = "significant", perf_metric_name = "balanced_acc", fair_metric_name = "equal_opportunity")                                                      

API functions

There are four main API functions that the user can execute to obtain the fairness diagnosis of their use cases.

Evaluate

The evaluate API function computes all performance and fairness metrics and renders it in a table format (default). It also highlights the primary performance and fairness metrics (automatic if not specified by user).

cre_sco_obj.evaluate()

Output:

You can also toggle the widget to view your results in a interactive visualization format.

cre_sco_obj.evaluate(visualize = True)

Output:

Tradeoff

Computes trade-off between performance and fairness.

cre_sco_obj.tradeoff()

Output:

Feature Importance

Computes feature importance of protected features using leave one out analysis.

cre_sco_obj.feature_importance()

Output:

Compile

Generates model artifact file in JSON format. This function also runs evaluate(), tradeoff() and feature_importance() if it hasn't already been ran.

cre_sco_obj.compile()

Output:

Model Artifact

A JSON file that stores all the results from evaluate(), tradeoff() and feature_importance().

Output:

License

Veritas Toolkit is licensed under the MIT License - see LICENSE for more details.