PyPI: classifier-toolkit

pip install classifier-toolkit

Classifier Toolkit

Table of Content

1. Installation
2. Usage
3. Modules Overview
4. Development & CI/CD
5. Future Work

Installation

This library is published in the PyPI directory. To install, users can run pip install 'classifier_toolkit' command.

Usage

This library automates binary classification tasks in the finance domain, specifically for default and fraud labeling. It includes several packages designed to address the main steps in any machine learning/data science task:

1. EDA: accessible via EDA_Toolkit. Provides EDA and feature engineering functionality with all necessary visualizations.
2. Feature Reduction: filter-style pre-selection pipeline (expert rules, low variance, drift, predictive power, counter-intuitive direction, high correlation).
3. Feature Selection: wrapper and embedded methods (RFE, Boruta, Sequential, Bayesian, ElasticNet, MetaSelector).
4. Model Training: accessible via Tuner. Hyperparameter optimization (grid search, Bayesian via Optuna) for LightGBM and CatBoost with train/val/test evaluation.
5. Evaluation and reporting: To be implemented.

For detailed usage, refer to the documentation.

Note: that this library does not contain data wrangling steps (although it contains feature engineering), it's an intermediate step between EDA and feature engineering where users should fix any data quality related issues. Therefore, conducting the EDA is crucial to mitigate any issues before moving onto the feature engineering and the subsequent steps.

Modules Overview

• EDA Toolkit: This module includes classes and methods for performing comprehensive exploratory data analysis. It provides automated warnings for data quality issues, univariate and bivariate analysis, and various data visualizations to help understand the dataset.

• Univariate Analysis: This class focuses on the analysis of individual variables. It includes methods for calculating statistical measures, visualizing distributions, and assessing relationships between variables and a target through techniques like Cramer's V and Information Value. This helps in understanding the significance and distribution of each feature independently.

• Bivariate Analysis: This class deals with the analysis of two variables to understand their relationship. It includes functionalities for generating correlation heatmaps, performing ANOVA tests between numerical and categorical variables, and computing pairwise Cramer's V for categorical features. This aids in identifying patterns and correlations between pairs of variables, which is crucial for feature selection and engineering.

• Feature Engineering: This module assists in transforming features, handling missing values, encoding categorical variables, and more. It aims to enhance the dataset's quality for better model performance.

• Visualizations: This module offers a wide range of plotting capabilities to visually analyze data distributions, relationships, and other crucial aspects of the dataset.

• Automated Warnings: A utility to automatically check the dataset for common issues such as missing or duplicate values, outliers, and more, providing warnings to guide data cleaning efforts.

• Feature Reduction: Filter-style pipeline that reduces the feature set before model-based selection. Six sequential steps, each independently configurable:

  1. Expert Rules — drop features by hand-coded list.
  2. Low Variance — drop near-constant numerical and categorical features.
  3. Drift — drop features whose distribution has shifted (PSI, KS, JS and more).
  4. Predictive Power — drop features with weak univariate Gini / PR-AUC.
  5. Counter-Intuitive Direction — drop features violating a business prior.
  6. High Correlation — drop pairwise-redundant features (Pearson/Spearman/Kendall; Cramér's V).

  The FeatureReducer orchestrates all six steps in one sklearn-compatible fit / transform call.

  from classifier_toolkit.feature_reduction import FeatureReducer
  
  reducer = FeatureReducer(gini_threshold=0.01, numeric_correlation_threshold=0.85)
  reducer.fit(X_train, y=y_train, X_val=X_val)
  X_filtered = reducer.transform(X_train)
  
  reducer.summary()                          # print pipeline summary
  reducer.export_summary("summary.xlsx", format="excel")
  

  Standalone helpers are also available for ad-hoc analysis outside the pipeline:

  from classifier_toolkit.feature_reduction import (
      calculate_feature_predictive_metrics,  # Gini + PR-AUC for every feature
      CorrelationAnalyser,                   # inspect all correlated pairs
      DriftAnalyser,                         # inspect drift stats per feature
  )
  

  Logs are silent by default. To enable them:

  import logging
  
  logging.basicConfig(level=logging.INFO)
  logging.getLogger("classifier_toolkit.feature_reduction").setLevel(logging.INFO)
  

• Feature Selection: This module provides various feature selection techniques:

  • Embedded Methods: Includes ElasticNet for regularization-based feature selection.
  • Wrapper Methods:
    • Recursive Feature Elimination (RFE) with support for various ensemble methods (Random Forest, XGBoost, LightGBM, CatBoost).
    • Sequential Feature Selection (forward, backward, floating, and bidirectional).
    • Recursive Feature Eliminator: Adaptive two-phase elimination (batch drop + one-by-one) with overfitting-aware strategies, early stopping with rollback, temporal cross-validation, and sample weights.
    • Feature Combination Searcher: Exhaustive combinatorial search over all subsets of a candidate pool, scoring each combination on train/val/test splits.
  • Meta Selector: Combines multiple feature selection methods to provide a robust selection.
  • Utility Functions: Includes scoring functions and plotting utilities for feature importance visualization.

• Model Training: Hyperparameter optimization for LightGBM and CatBoost, with support for grid search and Bayesian optimization (via Optuna).

  from classifier_toolkit.model_training.hyper_parameter_tuning import Tuner
  
  tuner = Tuner(model_name="lightgbm", metric="auc", search_type="bayesian", n_trials=50)
  tuner.fit(X_train, y_train, X_val, y_val, X_test=X_test, y_test=y_test)
  
  best_model = tuner.best_model
  tuner.results_df  # full trial results
  

  Custom parameter search spaces can be defined via ModelParams and ParamRange.

Development & CI/CD

This project uses modern tooling for fast and efficient development workflows:

Dependency Management

• UV: Lightning-fast Python package installer and resolver (replacing Poetry)
• Install UV: curl -LsSf https://astral.sh/uv/install.sh | sh
• Install dependencies: uv sync --group dev --group lint --group test

CI/CD Pipeline

Our CI/CD pipeline is optimized for speed and efficiency:

• Parallel Test Execution: Tests are split into two groups (eda and feature_selection) that run simultaneously, reducing test time by ~50%
• Shared Caching: Both parallel jobs share the same dependency cache (~1.7GB), avoiding duplicate downloads
• Smart Test Reruns: Failed tests run first (pytest --lf --ff) for faster feedback on fixes
• Master Protection: Build tests only run on master branch and PRs targeting master, saving CI resources on feature branches
• Automatic Linting: Code quality checks (Ruff, SQLFluff) run on every push

Pipeline Jobs:

1. dependencies - Installs and caches project dependencies
2. lint - Runs code quality checks (Ruff, SQLFluff)
3. test - Executes tests in parallel with shared cache
4. build_test - Builds and validates package (master/PRs only)

Local Development:

# Run linting
uv run make lint

# Run tests
uv run make test

# Build package
uv build

Future Work

The next planned improvements and additions to the library include:

• Developing comprehensive evaluation and reporting tools to assist with model assessment.
• Expanding documentation to include architecture diagrams and detailed usage examples.