datacleanr 0.1.1

Clean messy real-world datasets with safe, explainable defaults before ML.

datacleaner

Clean messy real-world datasets with safe, explainable defaults before ML.

Overview

datacleaner is a practical data-cleaning library for tabular datasets in pandas. It is designed for ML practitioners, students, and data engineers who need a reliable baseline cleaning pipeline. The library emphasizes safety, transparent decisions, and clear diagnostics so you can inspect what changed and why.

Key Features

• Missing value handling with smart fill and high-missing column dropping.
• Duplicate removal for exact duplicate rows.
• Datatype correction with numeric and datetime detection.
• Text standardization (lowercase, trim, and common value normalization).
• Outlier handling with cap or remove modes, plus capping diagnostics.
• Column selection for constant columns and likely ID-like columns.
• Correlation reduction for highly correlated numeric features.
• Safety mode to prevent excessive row or column loss.
• Structured diagnostics reporting for explainable cleaning.

Quick Start

Minimal usage:

from datacleaner import clean

df = clean(df)

Advanced usage with report and verbose logs:

from datacleaner import clean

df, report = clean(df, return_report=True, verbose=True)

Example Output

Missing values: dropped 1 columns, filled 99 values.
Duplicates: removed 5 rows.
Outliers (cap): capped 12 values.
Column selection: dropped 1 low-information columns.

Diagnostics:
- numeric_candidate_columns: ['mixed_numeric']
- max_correlation: 0.48
- high_outlier_capping_columns: []

Diagnostics and Safety

• numeric_candidate detection highlights text columns that are mostly numeric-like.
• Outlier diagnostics flag columns with high capping rates to avoid silent over-correction.
• Correlation diagnostics report max observed correlation and number of pairs checked.
• safe_mode=True warns when data loss is high and can roll back aggressive steps.

When to Use / When Not to Use

Use datacleaner when:

• You are working with messy real-world tabular datasets.
• You need a quick, safe baseline cleaning pass before modeling.

Do not use datacleaner when:

• You need full ML preprocessing (for example scaling, encoding, or feature engineering).
• Your workflow requires domain-specific cleaning rules that cannot be generalized.

Installation

pip install datacleaner

Testing

The library has been validated using:

• synthetic stress tests
• sklearn datasets
• edge-case scenarios

Run tests:

python tests/test_sklearn_datasets.py

Project Structure

datacleaner/
	src/datacleaner/
		core.py
		analysis.py
		missing_values.py
		duplicates.py
		datatypes.py
		text_standardization.py
		outliers.py
		column_selection.py
		correlation_reduction.py
		reporting.py
	tests/
	examples/

Contributing

Contributions are welcome. Please open an issue for bug reports or design proposals, and submit a pull request for fixes and improvements.