probatus 0.1.2

Tools for machine learning model validation

# Probatus

## Overview

Library for the validation of binary classifiers

## Installation

`bash pip install probatus `

## Features

#### 1 Metrics for distirbution similarity check Everything related to metrices used for the validation of the models. On top of the usual metrics defined in sklearn, provide: - [ ] PSI score - [ ] KS values - [ ] Hypothesis testing for distribution similarity - [ ] Something else the smart data scientists will think about ;)

#### 2 Metrics uncertainty This part is often underestimated. With unbalanced dataset and few cases of low there is a big issue as some. - [ ] Resampling n_times (at least 100) of the train test split to get distirbution of the metrices (example AUC) - [ ] Implement it in a parallel fashion (n_jobs) to speed it up - [ ] Smart way to correct for correlated indipendent folds

#### 3 Bucketing strategy Some metrices require different binning algorithms to be computed (for example, PSI). This epic covers multiple algorithms related to the way how customers are grouped together used for risk bucketing calculations - [ ] Implement simple binning - [ ] Implement one dimensional clustering based on model prediction - [ ] Implement the “Optyparm” algorithm - [ ] Think of different bucketing algorithms

#### 4 Model calibration Go beyond the isotonic regression for calibration - implement likelihood estimates with parametrizable functionality - [ ] Implement fitting of “logistic regression” with non linear dependencies between the expected predicitons and the actual defatul rates per risk bucket

#### 5 Sample validation Checks related to the validation of the samples definition. This includes building propensity models for the sample definiton - [ ] Build different propensity models to validate the similarity of the samples - [ ] Perform univarite checks of feature distributions over different samples - [ ] Correlations change between samples

#### 6 Effects of correlations between features - [ ] Think about it

#### 7 Time stability Evaluate the stability of the model predictions over time - [ ] Build different propensity models to validate the similarity of time samples - [ ] Perform univarite checks of feature distributions over different samples - [ ] Calculate Shapley values over different time intervals and try to compare distirbutions with metrices define in epic 1 - [ ] Evaluate the stability of the model predictions in differnet time subsamples with metrices defined in epic 1 (to be used as model performance metric) - [ ] Something else?

#### 8 Interpretability Basically shapely values - [ ] Extend this

#### 9 Features impact on models This includes checks like permutation importance - [ ] Implement permutation importance - [ ] Define a strategy to estimate the systematic uncertainty of the feature (to be used for monitoring model performance)

#### 10 Model version control Implement the Crayon way of autotagging the models if developed in a git repo - [ ] Implement the current version control mechanism from Crayon

#### 11 Reporting Make a simple reporting tool that generates a standrazie document with the most important findings - [ ] Think about this - to be refined

#### 12 Bias detection This one is hard - [ ] Think about this - to be refined abd rethought

#### 13 Host it on a server Make a flask app that enables a server like hosting (something like jupyter) - [ ] Build a simple Flask Server - [ ] Implement a command line interace (CLI) to run the flask app and lounche a web app (jupyter like)

#### 14 Front end Related to the previous story, make a front end (maybe in Shiny?) to actually allow the modelers/validators/business people without any pyton knowledge to use the capabilities of the library (kind of import your data, your model and get everything in a clickable UI)

#### 15 Define some criteria for model quality Define some standard criteria for model quality that will give a quality score based on all the checks developed above

### 16 Feature interactions Factorization machines based approach - polynomial factorization regression

### Sample dataset is provided by the library

Sample test loan default data from Lending Club https://www.lendingclub.com/. Only a sample of all loans and available features is provided.

Metadata:

• id object - Loan ID
• loan_issue_date datetime64[ns] - Date when loan was issued. All dates are set to fist day of the month

• default int64 - Flag if the loan went to default or not.

  Default is defined as one of statuses: ‘Late (31-120 days)’, ‘Late (16-30 days)’, ‘Default’

• loan_amnt float64 - Amount requested
• funded_amnt float64 - Amount granted
• term int64 - Term of the loan in months
• int_rate float64 - Annual interest rate
• annual_inc float64 - Annual income of the client
• fico_range_low float64 - Fico rate range low
• fico_range_high float64 - Fico rate range high
• long_emp int64 - Indicator if client is employed for longer than 5 years
• credit_grade_A uint8 - Indicator if clients credit grade is A
• credit_grade_B uint8 - Indicator if clients credit grade is B
• credit_grade_C uint8 - Indicator if clients credit grade is C
• credit_grade_D uint8 - Indicator if clients credit grade is D
• credit_grade_E uint8 - Indicator if clients credit grade is E
• credit_grade_F uint8 - Indicator if clients credit grade is F
• credit_grade_G uint8 - Indicator if clients credit grade is G