PyPI: classifier-toolkit

pip install classifier-toolkit

Classifier Toolkit

Table of Content

1. Installation
2. Usage
3. Modules Overview
4. Development & CI/CD
5. Future Work

Installation

This library is published in the PyPI directory. To install, users can run pip install 'classifier_toolkit' command.

Usage

This library automates binary and multiclass classification workflows. It is independent of the modelled problem: the class of interest is configured through pos_label (the positive class for binary targets, the class of interest for multiclass ones). It includes several packages designed to address the main steps in any machine learning/data science task:

1. EDA: accessible via EDAToolkit. Provides EDA and feature engineering functionality with all necessary visualizations.
2. Feature Reduction: filter-style pre-selection pipeline (expert rules, low variance, drift, predictive power, counter-intuitive direction, high correlation).
3. Feature Selection: wrapper and embedded methods (RFE, Boruta, Sequential, Bayesian, ElasticNet, MetaSelector).
4. Model Training: accessible via Tuner. Hyperparameter optimization (grid search, Bayesian via Optuna) for LightGBM and CatBoost with train/val/test evaluation, including multiclass objectives and class weights.
5. Risk Class: RiskClassBuilderDP / RiskClassBuilder turn model scores into risk classes with statistically validated, ordered event rates.
6. Model Validation: Evaluator computes metrics, calibration and stability checks for any model exposing predict_proba.
7. Calibration: OVRHistogramCalibrator and plot_reliability_curves for One-vs-Rest probability calibration.
8. Explainability: SHAP-based explanations, interactions and misclassification diagnostics.
9. Data Partition: temporal-aware train/test splitting, preprocessing and dtype optimization.

For detailed usage, refer to the documentation.

Note: that this library does not contain data wrangling steps (although it contains feature engineering), it's an intermediate step between EDA and feature engineering where users should fix any data quality related issues. Therefore, conducting the EDA is crucial to mitigate any issues before moving onto the feature engineering and the subsequent steps.

Modules Overview

• EDA Toolkit: This module includes classes and methods for performing comprehensive exploratory data analysis. It provides automated warnings for data quality issues, univariate and bivariate analysis, and various data visualizations to help understand the dataset.

• Univariate Analysis: This class focuses on the analysis of individual variables. It includes methods for calculating statistical measures, visualizing distributions, and assessing relationships between variables and a target through techniques like Cramer's V and Information Value. This helps in understanding the significance and distribution of each feature independently.

• Bivariate Analysis: This class deals with the analysis of two variables to understand their relationship. It includes functionalities for generating correlation heatmaps, performing ANOVA tests between numerical and categorical variables, and computing pairwise Cramer's V for categorical features. This aids in identifying patterns and correlations between pairs of variables, which is crucial for feature selection and engineering.

• Feature Engineering: This module assists in transforming features, handling missing values, encoding categorical variables, and more. It aims to enhance the dataset's quality for better model performance.

• Visualizations: This module offers a wide range of plotting capabilities to visually analyze data distributions, relationships, and other crucial aspects of the dataset.

• Automated Warnings: A utility to automatically check the dataset for common issues such as missing or duplicate values, outliers, and more, providing warnings to guide data cleaning efforts.

• Feature Reduction: Filter-style pipeline that reduces the feature set before model-based selection. Six sequential steps, each independently configurable:

  1. Expert Rules — drop features by hand-coded list.
  2. Low Variance — drop near-constant numerical and categorical features.
  3. Drift — drop features whose distribution has shifted (PSI, KS, JS and more).
  4. Predictive Power — drop features with weak univariate Gini / PR-AUC.
  5. Counter-Intuitive Direction — drop features violating a business prior.
  6. High Correlation — drop pairwise-redundant features (Pearson/Spearman/Kendall; Cramér's V).

  The FeatureReducer orchestrates all six steps in one sklearn-compatible fit / transform call.

  from classifier_toolkit.feature_reduction import FeatureReducer
  
  reducer = FeatureReducer(gini_threshold=0.01, numeric_correlation_threshold=0.85)
  reducer.fit(X_train, y=y_train, X_val=X_val)
  X_filtered = reducer.transform(X_train)
  
  reducer.summary()                          # print pipeline summary
  reducer.export_summary("summary.xlsx", format="excel")
  

  Standalone helpers are also available for ad-hoc analysis outside the pipeline:

  from classifier_toolkit.feature_reduction import (
      calculate_feature_predictive_metrics,  # Gini + PR-AUC for every feature
      CorrelationAnalyser,                   # inspect all correlated pairs
      DriftAnalyser,                         # inspect drift stats per feature
  )
  

  Logs are silent by default. To enable them:

  import logging
  
  logging.basicConfig(level=logging.INFO)
  logging.getLogger("classifier_toolkit.feature_reduction").setLevel(logging.INFO)
  

• Feature Selection: This module provides various feature selection techniques:

  • Embedded Methods: Includes ElasticNet for regularization-based feature selection.
  • Wrapper Methods:
    • Recursive Feature Elimination (RFE) with support for various ensemble methods (Random Forest, XGBoost, LightGBM, CatBoost).
    • Sequential Feature Selection (forward, backward, floating, and bidirectional).
    • Recursive Feature Eliminator: Adaptive two-phase elimination (batch drop + one-by-one) with overfitting-aware strategies, early stopping with rollback, temporal cross-validation, and sample weights.
    • Feature Combination Searcher: Exhaustive combinatorial search over all subsets of a candidate pool, scoring each combination on train/val/test splits.
  • Meta Selector: Combines multiple feature selection methods to provide a robust selection.
  • Utility Functions: Includes scoring functions and plotting utilities for feature importance visualization.

• Model Training: Hyperparameter optimization for LightGBM and CatBoost, with support for grid search and Bayesian optimization (via Optuna).

  from classifier_toolkit.model_training.hyper_parameter_tuning.tuner import Tuner
  
  tuner = Tuner(
      X=X_train, y=y_train,
      model_name="lightgbm",
      X_val=X_val, y_val=y_val,
      X_test=X_test, y_test=y_test,
      search_method="bayesian",
      n_trials=50,
      optimization_metric="prauc",
  )
  result = tuner.tune()
  
  best_model = result["best_model"]
  result["trials_results"]  # full trial results (DataFrame)
  

  Reported metrics are auc, prauc, ks, log_loss and brier (ks/brier are binary-only). Custom parameter search spaces can be defined via ModelParams and ParamRange.

• Risk Class: Builds risk classes from model scores. RiskClassBuilderDP searches bin edges with dynamic programming so that each class' observed event rate falls in a target band (target_ranges, required), then checks that adjacent classes are statistically distinguishable; RiskClassBuilder discovers classes with KMeans.

  from classifier_toolkit.risk_class import RiskClassBuilderDP
  
  builder = RiskClassBuilderDP(
      target_col="target",
      target_ranges=[(0.00, 0.02), (0.02, 0.05), (0.05, 0.10)],
      min_obs_per_bin=200,
  )
  result = builder.build(train_proba, y_train)
  print(result.bins, result.n_classes)
  

• Model Validation: Evaluator evaluates any model with predict_proba (metrics, ROC/PR/calibration/threshold plots), score_distribution_psi and compare_score_distributions compare a reference score distribution with a current one, and evaluate_risk_classes_dp checks calibration within each risk class.

• Calibration: OVRHistogramCalibrator calibrates multiclass (or binary) probabilities One-vs-Rest with histogram binning; plot_reliability_curves shows raw vs calibrated reliability per class.

• Explainability: TreeSHAPExplainer, ExplainabilityToolkit, SHAP plots, pairwise interaction analysis and a MisclassificationAnalyzer that explains confusion-matrix quadrants (per-class SHAP for multiclass models).

• Data Partition: Temporal-aware train/test splitting, preprocessing helpers and dtype optimization.

Development & CI/CD

This project uses modern tooling for fast and efficient development workflows:

Dependency Management

• UV: Lightning-fast Python package installer and resolver (replacing Poetry)
• Install UV: curl -LsSf https://astral.sh/uv/install.sh | sh
• Install dependencies: uv sync --group dev --group lint --group test

CI/CD Pipeline

Our CI/CD pipeline is optimized for speed and efficiency:

• Parallel Test Execution: One test job per directory under tests/ (discovered automatically, so new test groups are picked up without editing the workflow), all running simultaneously
• Shared Caching: The parallel jobs share the same dependency cache, avoiding duplicate downloads
• Smart Test Reruns: Failed tests run first (pytest --lf --ff) for faster feedback on fixes
• Master Protection: Build tests only run on master branch and PRs targeting master, saving CI resources on feature branches
• Automatic Linting: Code quality checks (Ruff, SQLFluff) run on every push

Pipeline Jobs:

1. dependencies - Installs and caches project dependencies
2. lint - Runs code quality checks (Ruff, SQLFluff)
3. test - Executes tests in parallel with shared cache
4. build_test - Builds and validates package (master/PRs only)

Local Development:

# Run linting
uv run make lint

# Run tests
uv run make test

# Build package
uv build

Future Work

The next planned improvements and additions to the library include:

• Extending the evaluation and reporting tools in model_validation (more checks and report formats).
• Expanding documentation to include architecture diagrams and detailed usage examples.