Datascrubber

A data cleaning, preprocessing and visualisation toolkit for data science projects.

Datascrubber wraps the tedious parts of a data-science workflow — reading messy files, fixing types, handling missing values and outliers, encoding and scaling for modelling, checking data quality, and visualising relationships — behind one class with a consistent, chainable API. Every cleaning action is logged, and fit-based transforms can be saved as a recipe and replayed on new data without leakage.

• Author: Charles Muganga
• Licence: MIT
• Python: 3.10+

Table of contents

• Installation
• Quickstart
• Core concepts
• Reading and writing data
• Profiling and quality
• Cleaning: missing values, duplicates, outliers
• Preprocessing for modelling
• Reproducible recipes and scikit-learn
• Class imbalance
• Visualisation
• The one-call pipeline
• Project structure
• Method reference
• Testing

Installation

From the project root (the folder containing pyproject.toml):

pip install Datascrubber

Quickstart

from Datascrubber import Datacleaning

dc = Datacleaning()
dc.read_data("Datasets/DiamondPricesData.xlsx")   # auto-describes the file

dc.auto_clean()          # standardise names, dedupe, impute, clip outliers
dc.savedata()            # -> cleaned_data.xlsx

auto_clean() prints a short summary:

Cleaning complete:
- Missing values have been handled (20 filled).
- Outliers have been handled.
- 146 duplicate row(s) removed.
- Column names standardised.

Call report() for the detailed, per-column log.

Everything is chainable (methods return self) and logged (dc.report() gives the full per-column detail).

Core concepts

One object, shared state. A Datacleaning instance holds your dataframe and every operation acts on it in place. Internally the class is composed of small focused mixins, but the public surface is a single flat API.

Fluent chaining. Mutating methods return self:

dc.standardize_columns().drop_duplicates().impute().remove_outliers()

Action log. Every change is recorded. dc.report() returns the detailed log; dc.diff(before) summarises what changed between two snapshot()s.

Notebook-friendly returns. Plots and pipelines return self (not a raw figure or string) so notebooks don't double-render images or echo escaped text. Pass show=False to a plot to get the figure back for saving/embedding instead.

Reading and writing data

dc.read_data("data.csv")                       # csv, xlsx, xls, json, txt, parquet, feather
dc.read_data("book.xlsx", sheet_name="Sheet2") # pick a sheet (name or index)
dc.read_url("https://example.com/data.csv")    # straight from a URL
dc.read_sql("SELECT * FROM sales", engine)      # any pandas-compatible connection

dc.savedata()                 # mirrors the input format -> cleaned_data.<ext>
dc.savedata("out.parquet")    # or choose a path/format explicitly

Multi-sheet Excel files no longer block on a prompt: with no sheet_name given, the first sheet is read and a note tells you how to pick another.

Profiling and quality

dc.summary()          # describe(include="all")
dc.columns()          # column index
dc.missing_values()   # per-column missing counts (+ a missingno matrix)
dc.data_types()       # split into categorical vs continuous

dc.smart_types()      # coerce "1,234" -> number, "yes"/"no" -> bool, dates -> datetime

dc.health_check()     # 0-100 quality score + flagged issues

health_check() flags missing data, duplicates, constant / quasi-constant columns, high-cardinality columns, and mixed-type columns, and returns a dict:

{"score": 99.7, "rows": 53940, "cols": 10, "issues": ["146 duplicate row(s)."]}

Schema validation checks your data against expectations:

dc.validate_schema({
    "price": {"dtype": "int", "min": 0},
    "cut":   {"allowed": ["Ideal", "Premium", "Very Good", "Good", "Fair"]},
})

Cleaning

Missing values — impute() supports several strategies:

dc.impute()                              # 'auto': mode for categoricals; for
                                         # numerics, median if skewed else mean
dc.impute(strategy="median")
dc.impute(strategy="knn")                # KNN imputation (numeric columns)
dc.impute(strategy="ffill")              # forward fill
dc.impute(strategy="median", group_by="region")   # group-wise
dc.drop_missing_values()                 # or just drop rows with NaNs

Duplicates and structure:

dc.drop_duplicates()
dc.standardize_columns()                 # snake_case, lowercase, de-punctuated
dc.drop(["id", "notes"])
dc.rename("old", "new")
dc.remove_empty_columns()

Outliers — clip with the IQR method, or use the flexible detector:

dc.remove_outliers()                     # clip all numeric columns to IQR fences

dc.detect_outliers("price", method="zscore", action="flag")    # add 'is_outlier'
dc.detect_outliers(method="iqr", action="remove")              # drop outlier rows
dc.detect_outliers(method="isolation_forest", action="flag")   # multivariate

method ∈ iqr, zscore, modified_zscore, isolation_forest; action ∈ flag, clip, remove.

Datetime features:

dc.parse_dates("order_date")
dc.extract_date_features("order_date")   # _year, _month, _day, _dayofweek,
                                         # _quarter, _is_weekend

Preprocessing for modelling

Encoding categorical columns:

dc.encode("cut", method="onehot")                    # dummy columns
dc.encode("grade", method="ordinal",
          ordering={"grade": ["low", "mid", "high"]})
dc.encode("city", method="frequency")                # replace with frequency
dc.encode("city", method="target", target="price")   # smoothed target mean
dc.encode(["a", "b"], method="label")

Scaling and binning:

dc.scale(["price", "carat"], method="standard")      # standard / minmax / robust
dc.scale(method="minmax")                             # all numeric columns
dc.bin_column("age", bins=4, strategy="quantile")    # -> age_binned

Reproducible recipes and scikit-learn

Fit-based transforms (impute, encode, scale) record the parameters they learn. Save that recipe and replay it on a test set to get identical transformations without leakage (no re-fitting on test data):

# --- fit on training data ---
train = Datacleaning(); train.dataframe = X_train
train.impute(strategy="median") \
     .encode("cut", method="onehot") \
     .scale(["price", "carat"], method="standard")
train.save_recipe("recipe.json")

# --- apply the SAME learned params to test data ---
test = Datacleaning()
test.load_recipe("recipe.json").apply_recipe(X_test)
X_test_clean = test.getdata()

Or use the scikit-learn transformer in a normal Pipeline:

from Datascrubber import DatascrubberTransformer
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

pre = DatascrubberTransformer(
    impute=True,
    scale="standard",
    encode={"columns": ["cut"], "method": "onehot"},
)

model = Pipeline([("prep", pre), ("clf", LogisticRegression())])
model.fit(X_train, y_train)      # prep fits on train only
model.predict(X_test)            # prep transforms test with train-learned params

Class imbalance

dc.class_balance("target")                  # distribution + imbalance ratio
dc.balance("target", method="oversample")   # duplicate minority up to majority
dc.balance("target", method="undersample")  # sample majority down to minority
dc.balance("target", method="smote")        # synthesise minority (needs
                                            # imbalanced-learn; falls back to
                                            # oversampling if unavailable)

SMOTE requires all features to be numeric — encode categoricals first.

Visualisation

All plots render once inline and return self; pass show=False to get the matplotlib figure back for saving or embedding.

dc.distributions()          # histogram grid of continuous columns
dc.cat_dist()               # countplot grid of categorical columns
dc.corr_matrix()            # correlation heatmap
dc.outliers()               # boxplot grid

dc.cont_to_cont("price", "carat")     # scatter + correlation
dc.cat_to_cat("cut", "color")         # countplot + contingency + chi-square
dc.cont_to_cat("price", "cut")        # boxplot + one-way ANOVA
dc.lineplot("cut", "price")

fig = dc.corr_matrix(show=False)      # get the figure instead of showing it
fig.savefig("corr.png")

An HTML profile report bundles the dataset overview and cleaning log:

dc.profile_report("report.html")

The one-call pipeline

dc.auto_clean(
    impute_strategy="auto",   # any impute() strategy
    standardize=True,         # snake_case column names
    dedupe=True,              # drop duplicate rows
    handle_outliers=True,     # clip to IQR fences
    verbose=True,             # print the short summary
)

auto_clean() resets the log, runs the steps, prints a concise summary and returns self. The detailed per-column log is always available via report(). data_cleaning() is kept as a backward-compatible alias (impute + clip).

Method reference