pandasclean

A lightweight Python library for cleaning and optimizing pandas DataFrames. Built for data analysts and data scientists who want practical, no-fuss data cleaning with sensible defaults.

Quick Start

import pandas as pd
from pandasclean import auto_clean

df = pd.read_csv('your_data.csv')
df_clean, report = auto_clean(df)

That's it. One line cleans your entire DataFrame.

Features

• Outlier Detection & Handling (IQR) — Detect outliers using the IQR method and choose to report, drop, or cap them
• Outlier Detection & Handling (Z-score) — Detect outliers using the Z-score method and choose to report, drop, or cap them
• Memory Reduction — Automatically downcast numeric dtypes and convert low cardinality string columns to save memory
• NaN Handling — Drop, fill with mean/median, or supply custom fill values per column
• Auto Clean — One function that runs everything with sensible defaults

Installation

pip install pandasclean

Usage

Auto Clean

Runs all cleaning functions in the correct order with sensible defaults.

from pandasclean import auto_clean

df_clean, report = auto_clean(df)

For custom behaviour, use the individual functions directly.

Outlier Detection & Handling

from pandasclean import find_outliers

# Report outlier bounds without changing data
df, bounds = find_outliers(df, strategy='report')

# Drop rows containing outliers
df_clean, bounds = find_outliers(df, strategy='drop')

# Cap outliers to the nearest bound (Winsorization)
df_clean, bounds = find_outliers(df, strategy='cap')

# Target specific columns with a custom multiplier
df_clean, bounds = find_outliers(df, columns=['age', 'salary'], multiplier=3.0, strategy='cap')

Z-score Outlier Detection & Handling

from pandasclean import find_outliers_zscore

# Report outlier info without changing data
df, info = find_outliers_zscore(df, strategy='report')

# Drop rows containing outliers
df_clean, info = find_outliers_zscore(df, strategy='drop')

# Cap outliers to mean ± (threshold × std)
df_clean, info = find_outliers_zscore(df, strategy='cap')

# Target specific columns with a custom threshold
df_clean, info = find_outliers_zscore(df, columns=['age', 'salary'], threshold=2.0, strategy='cap')

NaN Handling

from pandasclean import handle_nan

# Report null counts and percentages without making changes
df, report = handle_nan(df, strategy='report')

# Drop rows with any NaN
df_clean, report = handle_nan(df, strategy='drop', axis='rows', how='any')

# Drop columns where more than 50% of values are NaN
df_clean, report = handle_nan(df, strategy='drop', axis='columns', threshold=50)

# Fill NaN with column mean (numeric columns only)
df_clean, report = handle_nan(df, strategy='mean')

# Fill NaN with column median (numeric columns only)
df_clean, report = handle_nan(df, strategy='median')

# Fill all NaN with a single custom value
df_clean, report = handle_nan(df, strategy='custom', fill_value=0)

# Fill NaN with different values per column
df_clean, report = handle_nan(df, strategy='custom', fill_value={
    'age': 0,
    'name': 'unknown',
    'salary': 50000
})

Memory Reduction

from pandasclean import reduce_memory

# Optimize all columns with default settings
df_optimized, report = reduce_memory(df)

# Disable category conversion for string columns
df_optimized, report = reduce_memory(df, convert_category=False)

# Custom cardinality threshold for category conversion
df_optimized, report = reduce_memory(df, cardinality_threshold=0.3)

Benchmarks

Tested on a 1.5 million row DataFrame with mixed dtypes:

before = df.memory_usage(deep=True).sum() / (1024 * 1024)
df_optimized, report = reduce_memory(df)
after = df_optimized.memory_usage(deep=True).sum() / (1024 * 1024)

# Before: 152.71 MB
# After:  37.19 MB
# Reduction: 75.6%

Always use memory_usage(deep=True) for accurate measurement — pandas default undercounts string column memory.

How It Works

Memory Reduction

Outlier Detection

Uses the IQR method to compute bounds:

• lower_bound = Q1 - (multiplier × IQR)
• upper_bound = Q3 + (multiplier × IQR)

Standard multiplier values:

• 1.5 — mild outliers (default)
• 3.0 — extreme outliers only

Z-score Outlier Detection

Uses the Z-score method to compute bounds:

• z_score = (value - mean) / std
• lower_bound = mean - (threshold × std)
• upper_bound = mean + (threshold × std)

Standard threshold values:

• 2.0 — aggressive (catches more outliers)
• 3.0 — conservative (default, catches extreme outliers only)

Parameters

auto_clean(df)

find_outliers(df, columns, multiplier, strategy)

find_outliers_zscore(df, columns, threshold, strategy)

handle_nan(df, columns, strategy, fill_value, axis, how, threshold)

reduce_memory(df, columns, convert_category, cardinality_threshold)

Use Cases

• Data cleaning before analysis — handle outliers and NaN values in one step
• Memory optimization before ML training — reduce DataFrame size before fitting models, especially useful for GPU training where float32 is standard
• CSV/Parquet compression — smaller dtypes mean smaller files on disk. Saving to parquet after running reduce_memory can reduce file sizes significantly

Roadmap

• Outlier detection and handling (IQR method)
• Memory reduction (dtype downcasting + category conversion)
• NaN handling (drop, mean, median, custom)
• Auto clean
• Z-score based outlier detection
• Skewness detection and fixing
• Duplicate detection and removal
• HTML report generator

Contributing

We welcome contributions! If you want to help improve pandasclean, please consider the following:

1. Reporting Bugs: Open an issue on GitHub to report any bugs or any unexpected behavior.
2. Running Tests: Ensure that your changes are covered by tests. You can run them using:

   python -m unittest tests.py
   python -m unittest discover
   

3. Pull Requests: When submitting a PR, please provide a clear description of your changes and ensure that the tests pass.
4. Documentation: If you add a new feature, please also update the documentation.

License

MIT License