ml-processor 0.5.20

Includes functions for performing econometrics tasks

ml_processor is a library written in python for perfoming most of the common data preprocessing tasks involved in building machine learning models. It includes methods for:

• perfroming necessary data transformation desired for machine learning modesl

• hyperparameter tunning using different methods notably One Hot encoding and WOE transformation

• automted machine learning model fitting and model performacne evaluation

Table of Contents

• Installation

  • Depencies

• Getting started

  • Tutorials

  • Example: config

    • get_credentials

  • Example: eda_data_quality

  • Example: binary_eda_plot

  • Example: data_prep

    • One Hot Encoding

      • create_encoder

      • oneHot_transform

    • WoE transformation

      • woe_bins

      • woe_bin_table

      • get_var_bins

      • woe_transform

      • woe_features

      • balance_data

  • Example: xgbmodel

    • model_results

    • model_results

Installation

To install the latest release of ml_processor from PyPi:

pip install ml_processor

Depencies

ml-processor requires

• pandas

• numpy

• matplotlib

• seaborn

• logging

• json

• dotenv

• sklearn

• optbinning

• pickle

• joblib

• snowflake

• sqlalchemy+

• xgboost

• statsmodels

• hyperopt

• scipy

• shap

• lightgbm

Getting started

Tutorials

Example: config

config from the configuration sub-module provides a conveneient way for working with information such as credentials that one might want to keep secret and not include into their script. It also provides an easy of logging information both to the console and creating of log files.

get_credentials

Takes as an argument a path to the location of a stored .env file and returns the contents in the file.

from ml_processor.configuration import config

>>> config.get_credentials('./examples/.env')

OrderedDict([('username', 'email.example.com'), ('password', 'myPassword')])

Example: eda_data_quality

Checks dataset aganist specific rules and assigns a data quality score.

Let us load the Home Credit Default Risk dataset provided on kaggle and perform qaulity checks on it

import pandas as pd

df = pd.read_csv('./data/application_train.csv')

from ml_processor.eda_analysis import eda_data_quality

>>> eda_data_quality(df).head()

2022-10-03 23:15:19,318:INFO:rule_1 : More than 50% of the data missing
2022-10-03 23:15:19,319:INFO:rule_2 : Missing some data
2022-10-03 23:15:19,319:INFO:rule_3 : 75% of the data is the same and equal to the minimum
2022-10-03 23:15:19,319:INFO:rule_4 : 50% of the data is the same and equal to the minimum
2022-10-03 23:15:19,320:INFO:rule_5 : Has negative values
2022-10-03 23:15:19,320:INFO:rule_6 : Possible wrong data type

                             type  unique  missing pct.missing      mean  min  25%  50%     75%  max  rule_1  rule_2  rule_3  rule_4  rule_5  rule_6  quality_score
elevators_mode            float64      26   163891       53.3%  0.074490  0.0  0.0  0.0  0.1208  1.0       1       1       0       1       0       1       0.400000
nonlivingapartments_avg   float64     386   213514       69.4%  0.008809  0.0  0.0  0.0  0.0039  1.0       1       1       0       1       0       0       0.528571
elevators_avg             float64     257   163891       53.3%  0.078942  0.0  0.0  0.0  0.1200  1.0       1       1       0       1       0       0       0.528571
nonlivingapartments_mode  float64     167   213514       69.4%  0.008076  0.0  0.0  0.0  0.0039  1.0       1       1       0       1       0       0       0.528571
elevators_medi            float64      46   163891       53.3%  0.078078  0.0  0.0  0.0  0.1200  1.0       1       1       0       1       0       0       0.528571

We pass data and generate the quality score for all the columns in the data.

Example: binary_eda_plot

Visualizes the distribution of labels of a binary target variable within each attribute of the different characteristics (features). For categorical variables, each categorical level is an attribute while for numerical variables, the attributes are created by splitting the variable at different percentiles with each group having 10% of the total data. If the value is the same at different percentiles, on the maximum percentile is considered and all the values upto that percentile assigned the same attribute.

We again use the Home Credit Default Risk dataset and plot a few columns.

First we initiate the plots by passing the dataset. If we want to plot specific columns, we pass plot_columns; a dict with variables grouped by their data types e.g {'target': [string name of target column], 'discrete' : [list of discrete columns], 'numeric': [list of numeric columns]}. Incase of columns that should use logarithmic scale, we pass log_columns; alist of columns to use logarithmic scale.

In this example, we simply pass the data and keep the default for the other parameters since we want to plot all columns and we don;t want to have any logarithmoc scales. We also use the default palette {1:'red', 0:'deepskyblue'}; you can change to suit you need.

from ml_processor.eda_analysis import binary_eda_plot

# initiate plots
eda_plot = binary_eda_plot(df_plot)

# generate plots
>>> eda_plot.get_plots()

After the plots ahve been initiated, we call the get_plots method to generate the plots.

Example: data_prep

data_prep provides a conevient way for transforming data into formats that machine learning models can work with more easily

We initiate the data_prep by passing the data, features and the categories

# define the variables
target = 'target'
features = ['amt_income_total', 'name_contract_type','code_gender']
categories = ['name_contract_type','code_gender']

from ml_processor.data_prep import data_prep

# initiate data transformation
init_data = data_prep(data=df_transform, features=features, categories=categories)

Two types of transfromation are currently possible:

• One Hot Encoding - For some more details, Jason Brownlee covers Why One Hot Encode Data in Machine Learning.

• Weight of Evidence (WoE) Transformation - Some futher details on the The binning procedure.

One Hot Encoding

create_encoder

Create One Hot encoder. Running this method creates a sub-directory data_prep within the cureent working working directory and saves the created encoder as a pickle file encoder. The saved encoder can be then load as pickle file and used to transform data in othern enviroments like production

encoder = init_data.create_encoder()

oneHot_transform

Calling oneHot_transform transforms the data using the encoder created using create_encoder method. If the encoder has not yet been created, calling oneHot_transform triggers the creation and saving of the encoder first using the create_encoder.

df_encode = init_data.oneHot_transform()
>>> df_encode.head()

   target  amt_income_total name_contract_type code_gender  name_contract_type_Revolving loans  code_gender_M
0       0          315000.0         Cash loans           M                                 0.0            1.0
1       0          382500.0         Cash loans           F                                 0.0            0.0
2       0          450000.0         Cash loans           M                                 0.0            1.0
3       0          135000.0         Cash loans           M                                 0.0            1.0
4       0           67500.0         Cash loans           M                                 0.0            1.0

You can obtain the encoder using the encoder property.

>>> init_data.encoder

OneHotEncoder(drop='if_binary', handle_unknown='ignore', sparse=False)

WoE transformation

The WoE transformation executes several methods from optbinning provided by Guillermo Navas-Palencia. Further details can be found on github OptBinning.

woe_bins

Generate binning process for woe transformation. The binning process created is saved as binningprocess.pkl in the sub-directory data_prep in the current working directory

init_data.woe_bins()

To get the created binning process created, use the property **binning_process**

>>> init_data.binning_process

BinningProcess(categorical_variables=['name_contract_type', 'code_gender',
                                   'flag_own_car', 'flag_own_realty',
                                   'name_type_suite', 'name_income_type',
                                   'name_education_type',
                                   'name_family_status', 'name_housing_type',
                                   'occupation_type',
                                   'weekday_appr_process_start',
                                   'organization_type', 'fondkapremont_mode',
                                   'housetype_mode', 'wallsmaterial_mode',
                                   'emergencystate_mo...
                            'name_type_suite', 'name_income_type',
                            'name_education_type', 'name_family_status',
                            'name_housing_type',
                            'region_population_relative', 'days_birth',
                            'days_employed', 'days_registration',
                            'days_id_publish', 'own_car_age', 'flag_mobil',
                            'flag_emp_phone', 'flag_work_phone',
                            'flag_cont_mobile', 'flag_phone', 'flag_email',
                            'occupation_type', 'cnt_fam_members',
                            'region_rating_client', ...])

woe_bin_table

Shows the summary results of the created bins.

bin_table = init_data.woe_bin_table()
>>> bin_table.head()

                name        dtype   status  selected n_bins        iv        js      gini quality_score
0       ext_source_3    numerical  OPTIMAL      True      6  0.419161  0.050627  0.351672      0.214852
1       ext_source_1    numerical  OPTIMAL      True      7  0.325791  0.039244  0.306015      0.185009
2       ext_source_2    numerical  OPTIMAL      True      7  0.278363  0.033828  0.286398      0.157844
3  organization_type  categorical  OPTIMAL      True      5  0.129885  0.015735  0.170484      0.280232
4      days_employed    numerical  OPTIMAL      True      5   0.10551  0.013074  0.176601      0.203093

get_var_bins

Shows the distribution of the classes within the bins created. We pass the variable whose bins we wish to see.

>>> init_data.get_var_bins('ext_source_3')

woe_transform

Transform data using the binning process created. If data is passed, it should have the same features as those used in fitting the binning process.

df_woe = init_data.woe_transform()
>>> df_woe.head()

   sk_id_curr  name_contract_type  amt_income_total  amt_credit  amt_annuity  amt_goods_price  name_education_type  name_family_status  region_population_relative  days_birth
0   -0.101520           -0.065406         -0.042766   -0.089406     0.021714        -0.121048             0.296993           -0.230281                    0.119906    0.224803
1   -0.138405            0.754275         -0.124456    0.035788     0.419484        -0.121048            -0.200188            0.100845                    0.119906   -0.015920
2   -0.138405           -0.065406         -0.042766   -0.089406     0.156751         0.316737             0.296993            0.100845                   -0.385415   -0.015920
3   -0.138405           -0.065406         -0.124456   -0.089406     0.021714        -0.121048            -0.200188           -0.230281                    0.119906   -0.015920
4   -0.101520           -0.065406         -0.042766   -0.089406     0.156751         0.316737            -0.200188            0.100845                    0.505606    0.224803

woe_features

Get features selected using the selection criteria defined during woe binning with woe_bins

woe_features = init_data.woe_features()
>>> woe_features

array(['code_gender', 'amt_credit', 'amt_annuity', 'amt_goods_price',
    'name_income_type', 'name_education_type',
    'region_population_relative', 'days_birth', 'days_employed',
    'days_registration', 'days_id_publish', 'flag_emp_phone',
    'occupation_type', 'region_rating_client',
    'region_rating_client_w_city', 'reg_city_not_work_city',
    'organization_type', 'ext_source_1', 'ext_source_2',
    'ext_source_3', 'apartments_avg', 'basementarea_avg',
    'years_beginexpluatation_avg', 'elevators_avg', 'entrances_avg',
    'floorsmax_avg', 'livingarea_avg', 'nonlivingarea_avg',
    'apartments_mode', 'basementarea_mode',
    'years_beginexpluatation_mode', 'elevators_mode', 'entrances_mode',
    'floorsmax_mode', 'livingarea_mode', 'nonlivingarea_mode',
    'apartments_medi', 'basementarea_medi',
    'years_beginexpluatation_medi', 'elevators_medi', 'entrances_medi',
    'floorsmax_medi', 'livingarea_medi', 'nonlivingarea_medi',
    'housetype_mode', 'totalarea_mode', 'wallsmaterial_mode',
    'emergencystate_mode', 'days_last_phone_change', 'flag_document_3'],
   dtype='<U28')

balance_data

Balance data based on target column such that each of the labels within the target classes has the same amount of data which is equal to the minimum size of the labels. If we pass data that is different from the one used when initiating data_prep, the new dataset should have the same target column name or the name of the new target columns should be passed as well.

df_balanced = init_data.balance_data(df_woe)

Here we balance a new data set different from the one used in intiating data_prep. The target column is however the same and we don’t pass any target. The results after balancing can be seen below:

fig, ax = plt.subplots(1,2, figsize=(8,4))

sns.countplot(x='target', data=df_woe, hue='target', dodge=False, ax=ax[0], palette=palette)

sns.countplot(x='target', data=df_balanced, hue='target', dodge=False, ax=ax[1], palette=palette)
ax[0].legend('', frameon=False)
ax[0].set_title('Unbalanced')

ax[1].legend('', frameon=False)
ax[1].set_title('Balanced')

plt.subplots_adjust(wspace=0.75)

plt.show()

Example: xgbmodel

Performing machine learning tasks including hyperparameter tuning and xgb model fitting.

First, we initiate the model fitting. We use the data transformed in the previous section uisng WoE transformation.

from ml_processor.model_training import xgbmodel

xgb_woe = xgbmodel(df_balanced
                ,features=woe_features
                ,target=target
                ,hyperparams=None
                ,test_size=0.25
                ,params_prop=0.1
                   )

model_results

Fitting model and performing model diagnostics.

>>> xgb_model = xgb_woe.model_results()

2022-10-03 10:05:55,337:INFO:Splitting data into training and testing sets completed
2022-10-03 10:05:55,338:INFO:Training data set:37237 rows
2022-10-03 10:05:55,339:INFO:Testing data set:12413 rows
2022-10-03 10:05:55,339:INFO:Hyper parameter tuning data set created
2022-10-03 10:05:55,340:INFO:Hyper parameter tuning data set:4965 rows
2022-10-03 10:05:55,346:INFO:Splitting hyperparameter tuning data into training and testing sets completed
2022-10-03 10:05:55,347:INFO:Hyperparameter tuning training data set:3723 rows
2022-10-03 10:05:55,347:INFO:Hyperparameter tuning testing data set:1242 rows
2022-10-03 10:05:55,348:INFO:Trials initialized...
100%|████████| 48/48 [00:26<00:00,  1.78trial/s, best loss: -0.7320574162679426]
2022-10-03 10:06:22,352:INFO:Hyperparameter tuning completed
2022-10-03 10:06:22,353:INFO:Runtime for Hyperparameter tuning : 0 seconds
2022-10-03 10:06:22,354:INFO:Best parameters: {'colsample_bytree': 0.3, 'gamma': 0.2, 'learning_rate': 0.01, 'max_depth': 11, 'reg_alpha': 100, 'reg_lambda': 10}
2022-10-03 10:06:22,354:INFO:Model fitting initialized...
2022-10-03 10:06:22,355:INFO:Model fitting started...
2022-10-03 10:06:24,334:INFO:Model fitting completed
2022-10-03 10:06:24,334:INFO:Runtime for fitting the model : 11 seconds
2022-10-03 10:06:24,338:INFO:Model saved: ./model_artefacts/xgbmodel_20221003100547.sav
2022-10-03 10:06:24,341:INFO:Dataframe with feature importance generated
2022-10-03 10:06:24,363:INFO:Predicted labels generated (test)
2022-10-03 10:06:24,381:INFO:Predicted probabilities generated (test)
2022-10-03 10:06:24,385:INFO:Confusion matrix generated (test)
2022-10-03 10:06:24,390:INFO:AUC (test): 73%
2022-10-03 10:06:24,395:INFO:Precision (test): 68%
2022-10-03 10:06:24,395:INFO:Recall (test): 67%
2022-10-03 10:06:24,396:INFO:F_score (test): 67%
2022-10-03 10:06:24,398:INFO:Precision and Recall values for the precision recall curve created
2022-10-03 10:06:24,401:INFO:True positive and negativevalues for the ROC curve created
2022-10-03 10:06:24,451:INFO:Recall and precision calculation for different thresholds (test) completed

model_results

Model Evaluation

>>> xgb_woe.make_plots()