DataCleaner

Automated data cleaning & standardization pipeline for ML projects.

DataCleaner takes raw CSV/Excel data and transforms it into production-ready ML features - handling nulls, encoding categories, selecting the best scaler per column, and packaging everything into a reusable inference pipeline.

Features

Quick Start

Install

pip install clean-data-ml

With optional extras:

pip install clean-data-ml[plot]       # visualization (matplotlib, seaborn)
pip install clean-data-ml[imbalance]   # SMOTE oversampling support
pip install clean-data-ml[all]         # all optional features

For a development (editable) install from source:

git clone https://github.com/MohammadvHossein/clean-data-ml.git
cd clean-data-ml
pip install -e .
pip install -e .[all]                # including all extras

Minimal example

from clean_data_ml import DataCleaner
from sklearn.svm import SVC

dc = DataCleaner()
dc.load("data.csv")
dc.set_target("purchased")
dc.drop_columns(["ID", "timestamp"])

X_train, X_test, y_train, y_test = dc.prepare(test_size=0.2)

model = SVC()
model.fit(X_train, y_train)
print(f"Accuracy: {model.score(X_test, y_test):.2f}")

Full Example

1. Train & save pipeline

from clean_data_ml import DataCleaner
import pandas as pd
from sklearn.svm import SVC
import joblib

# Sample data
data = pd.DataFrame({
    "ID": range(100),
    "age": [25, 30, 35, None, 40, 45, 50, 55, 60, 65] * 10,
    "salary": [50000, 60000, None, 80000, 90000, 100000, 110000, 120000, None, 140000] * 10,
    "city": ["Tehran", "Shiraz", "Tehran", "Isfahan", None, "Tehran", "Shiraz", "Isfahan", "Tehran", "Shiraz"] * 10,
    "gender": ["M", "F", "M", "F", "M", "F", "M", "F", "M", "F"] * 10,
    "purchased": [1, 0, 1, 0, 1, 1, 0, 1, 0, 1] * 10,
})

dc = DataCleaner()
dc.load_df(data)
dc.set_target("purchased")
dc.drop_columns(["ID"])

X_train, X_test, y_train, y_test = dc.prepare(test_size=0.2)

model = SVC(probability=True)
model.fit(X_train, y_train)
print(f"Accuracy: {model.score(X_test, y_test):.2f}")

# Save model & pipeline for later inference
joblib.dump(model, "model.pkl")
dc.save_pipeline("my_pipeline.pkl")

2. Inference on new data

from clean_data_ml import DataCleaner
import pandas as pd
from sklearn.svm import SVC
import joblib

dc = DataCleaner.load_pipeline("my_pipeline.pkl")
model = joblib.load("model.pkl")

new_data = pd.DataFrame({
    "age": [28, 42, 35],
    "salary": [65000, 95000, 78000],
    "city": ["Tehran", "Isfahan", "Shiraz"],
    "gender": ["F", "M", "F"],
})

processed = dc.transform(new_data)
predictions = model.predict(processed)
probabilities = model.predict_proba(processed)

for i, (pred, prob) in enumerate(zip(predictions, probabilities)):
    status = "Purchased" if pred == 1 else "Not Purchased"
    print(f"Customer {i+1}: {status} (confidence: {max(prob):.2%})")

API Reference

DataCleaner(random_state=42)

Main class. All methods return self for chaining.

prepare() Parameters

CleanPipeline (internal)

Holds all fitted transformers. Can be used directly, but prefer DataCleaner for full functionality.

Pipeline Attributes (accessible via dc.pipeline.*)

How Nulls Are Handled

The threshold for "drop vs impute" is dynamic -- it adapts to dataset size:

• Numeric columns with many nulls -- KNNImputer(n_neighbors=5)
• Categorical columns with many nulls -- filled with mode
• Any column with few nulls -- those rows are dropped

You can override this with prepare(null_drop_ratio=0.1).

Statistical Test Suite

The clean_data_ml.stats module provides a comprehensive set of statistical tests for data analysis:

Standalone Functions

StatisticalTestSuite (integration with DataCleaner)

from clean_data_ml import DataCleaner, stats

dc = DataCleaner()
dc.load_df(data).set_target("purchased")

suite = stats.StatisticalTestSuite(dc)
suite.test_normality()
suite.test_correlations(target_col="purchased")
suite.test_chi_square("gender", "city")
suite.test_anova("age", "city")
suite.test_z_one_sample("age", pop_mean=35)
suite.test_t_independent("age", "score")
suite.test_ab_by_group("converted", "group", "A", "B", metric_type="proportion")
print(suite.summary())

Visualization Module

The clean_data_ml.plotting module (requires pip install -e .[plot]):

Project Structure

clean_data_ml/
  __init__.py       Package exports
  cleaner.py        DataCleaner + CleanPipeline classes
  auto_scaler.py    Automatic scaler selection logic
  stats.py          Statistical test suite (t-test, z-test, AB test, etc.)
  plotting.py       Optional visualization module
setup.py                  Package metadata
pyproject.toml            Build configuration
MANIFEST.in               sdist inclusion rules
LICENSE                   MIT license
example_train.py          Training example
example_inference.py      Inference example
.pre-commit-config.yaml   Linting hooks (black, isort, flake8)
.gitignore                Ignored files
README.md                 This file
tests/
    conftest.py           Shared test fixtures
    test_cleaner.py       DataCleaner / CleanPipeline tests
    test_auto_scaler.py   Scaler selection tests
    test_stats.py         Statistical test suite tests
    test_plotting.py      Visualization module tests

Additional Features

Auto-Drop Useless Columns

The library automatically detects and removes:

• Zero-variance columns -- columns with a single unique value
• High-cardinality columns -- non-numeric columns where unique values exceed 90% of rows (e.g., free-text fields)

Disabled with prepare(auto_drop_useless=False).

Outlier Handling (IQR)

After null handling, each numeric column is checked using the Interquartile Range method:

• Lower bound: Q1 - 1.5 x IQR
• Upper bound: Q3 + 1.5 x IQR

Two modes:

• "clip" -- caps values at the bounds (preserves row count)
• "remove" -- drops rows with outliers

Activated with prepare(handle_outliers="clip").

Feature Engineering

Generates polynomial features (degree 2) for numeric columns with more than 2 unique values. Creates interaction terms and squared features automatically.

Activated with prepare(feature_engineering=True).

Date Feature Extraction

When prepare(extract_date_features=True), datetime columns are automatically expanded into numerical components:

• {col}_year, {col}_month, {col}_day, {col}_dayofweek, {col}_weekend
• The original datetime column is dropped afterward.

This happens early in the pipeline so the derived numeric columns benefit from all subsequent steps (encoding, scaling, feature engineering, etc.).

Missing Indicators

When prepare(add_missing_indicators=True), for every column that receives KNN imputation (null ratio above threshold), an additional binary column {col}_missing is added, flagging which rows originally contained nulls. This lets the model learn patterns from the missingness itself.

Feature Selection

Controlled by prepare(feature_selection="auto") or prepare(feature_selection=0.01).

After all transformations, Mutual Information is computed between each feature and the target. Features with MI below the threshold are dropped:

• "auto" -- drops features below the median MI score
• float (e.g., 0.01) -- drops features below that absolute threshold

Set to None (default) to skip feature selection entirely.

Custom Encoders & Custom Scalers

Pass fitted or unfitted sklearn-compatible transformers to override auto-detection:

from sklearn.preprocessing import OrdinalEncoder, KBinsDiscretizer

dc.prepare(
    custom_encoders={"city": OrdinalEncoder()},
    custom_scalers={"salary": KBinsDiscretizer(n_bins=5, encode="ordinal")},
)

These are stored in dc.pipeline.custom_encoders / dc.pipeline.custom_scalers and applied during transform() as well.

Imbalanced Data (SMOTE)

When handle_imbalance=True and the problem is classification, SMOTE oversampling is applied to the training set after the train/test split. Requires imbalanced-learn:

pip install imbalanced-learn

How Scaler Selection Works

For each numeric column, the library tests:

1. Normality -- Shapiro-Wilk test (p > 0.05 => normal)
2. Outliers -- IQR method (1.5x IQR rule)
3. Bounds -- min >= 0 & max <= 1
4. Sparsity -- >40% zeros

Note: Tree-based models (Random Forest, XGBoost, LightGBM, etc.) do not require scaling or normalization -- they split on thresholds and are invariant to monotonic transformations. Scaling is only needed for distance-based or gradient-based models (SVM, KNN, Neural Networks, Logistic Regression, etc.). You can skip auto-scaling with prepare(auto_scale=False) if using a tree-based model.

Then assigns the optimal scaler:

Requirements

• Python >= 3.8
• pandas >= 1.3
• numpy >= 1.21
• scikit-learn >= 1.0
• scipy >= 1.7
• joblib
• openpyxl (for Excel support)

License

MIT