Classic Machine Learning Framework

A dynamic, leakage-safe, production-oriented classic ML experimentation framework built primarily on scikit-learn.

It behaves like a lightweight AutoML / ML experiment runner for tabular data:

• Automatically inspects data
• Detects problem type (binary / multiclass classification, regression)
• Detects feature types (numeric, categorical, boolean, datetime, ID-like, text-like, constant, high-missingness)
• Builds reasoned preprocessing pipelines
• Runs staged model selection (baseline → candidates → shortlist → tune)
• Evaluates on a held-out test set once
• Produces error analysis, feature importance, and a full experiment report

Design priorities: correctness, no data leakage, reproducibility, strong baselines, explainable decisions, maintainability, extensibility.

---

Architecture

ml_framework/
├── main.py                 # CLI orchestration
├── configs/default.yaml
├── src/
│   ├── data/               # load, validate, profile, split
│   ├── detection/          # problem type, feature types, target, decision engine
│   ├── preprocessing/      # ColumnTransformer pipelines
│   ├── features/           # engineering / selection hooks
│   ├── models/             # extensible registry
│   ├── training/           # baseline, CV, tuning, final train
│   ├── evaluation/         # metrics, test eval, error analysis
│   ├── explainability/     # permutation (+ optional SHAP)
│   ├── experiments/        # runner + reporter
│   ├── persistence/        # joblib full-pipeline save/load
│   └── utils/              # logging, config, seeds
├── artifacts/              # models, reports, plots
└── tests/

Every major automatic choice is logged as:

Field	Meaning
Decision	What was chosen
Reason	Why
Action	Concrete effect
Confidence	high / medium / low

---

Installation

cd ml_framework
python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Python 3.11+ recommended.

---

Quick start

# Classification or regression — framework detects automatically
python main.py --data path/to/train.csv --target my_target

# Explicit problem type
python main.py --data train.csv --target SalePrice --problem-type regression

# External test set (Kaggle-style)
python main.py --train train.csv --test test.csv --target SalePrice

# Config file
python main.py --config configs/default.yaml --data train.csv --target y

# Disable tuning for a fast run
python main.py --data train.csv --target y --no-tuning

Predict

python main.py predict \
  --model artifacts/models/best_model.joblib \
  --data new_data.csv \
  --output predictions.csv

Profile only

python main.py profile --data train.csv --target y

---

Configuration

See configs/default.yaml. Important knobs:

• problem.type: auto | binary_classification | multiclass_classification | regression
• split.test_size, random_state
• preprocessing.high_cardinality_threshold, missing_threshold
• models.include / exclude
• tuning.enabled, method (randomized_search | grid_search), n_iter, shortlist_size
• evaluation.primary_metric: auto or sklearn scorer name / friendly alias (rmse, f1, roc_auc, …)
• explainability.enabled

CLI flags override YAML.

---

Supported problems & models

Problems: binary classification, multiclass classification, regression.

Models (registry): LogisticRegression, RidgeClassifier, DecisionTree, RandomForest, ExtraTrees, HistGradientBoosting, GradientBoosting, SVC, KNeighbors (classification); LinearRegression, Ridge, Lasso, ElasticNet, DecisionTree, RandomForest, ExtraTrees, HistGradientBoosting, GradientBoosting, SVR, KNeighbors (regression).

The Decision Engine selects a small candidate set based on dataset size and feature mix — it does not brute-force every model.

---

How leakage is prevented

1. Train/test split happens before any fit.
2. All imputation, scaling, encoding live inside an sklearn Pipeline + ColumnTransformer.
3. CV and tuning operate on the full pipeline (preprocess + model).
4. Final metrics are computed once on the untouched test set.
5. Model selection uses CV scores, never test scores.
6. External test sets are never used during training or tuning.

---

Extensibility

Add a model

from src.models.registry import register_model
from sklearn.ensemble import AdaBoostClassifier

register_model(
    "AdaBoostClassifier",
    AdaBoostClassifier,
    problem_types=["binary_classification", "multiclass_classification"],
    default_params={"random_state": 42},
    search_space={"n_estimators": [50, 100, 200]},
)

Add a data loader

from src.data.loader import register_loader

@register_loader("feather")
def load_feather(path):
    import pandas as pd
    return pd.read_feather(path)

Custom metric

Pass --metric my_scorer if registered with sklearn, or set evaluation.primary_metric in YAML.

---

Output structure

artifacts/
├── models/best_model.joblib    # full pipeline
├── reports/
│   ├── final_report.json
│   └── final_report.html
└── plots/
    ├── target_distribution.png
    ├── missing_heatmap.png
    ├── correlation_heatmap.png
    └── residuals.png           # regression

---

Example summary output

============================================================
ML EXPERIMENT COMPLETE
============================================================

Problem: Regression
Dataset: 1460 rows × 81 columns
Train: 1168 rows
Test: 292 rows
Primary Metric: RMSE
Baseline: -0.42
Best Model: HistGradientBoostingRegressor
CV: -0.251 ± 0.009
Test rmse: 0.237
Test mae: 0.164
Test r2: 0.891
Model saved: artifacts/models/best_model.joblib
Report: artifacts/reports/final_report.html
============================================================

---

Limitations

• Classic tabular ML only (no deep learning, no raw text/image models).
• Text-like columns are detected and dropped with a clear message.
• Very high-cardinality categoricals use OrdinalEncoder (not target encoding) to avoid leakage.
• Bayesian optimization / SHAP are optional extras.
• Not a guarantee of the globally optimal model — it aims for strong, reproducible baselines with transparent decisions.

---

Tests

cd ml_framework
pytest tests/ -q

---

License

MIT-style — use freely in research and production prototypes.