cleanix 0.1.0

Intelligent, explainable data cleaning built on pandas.

Cleanix

Intelligent, explainable data cleaning built on pandas.

Cleanix automatically detects and fixes common data-quality problems — missing values, duplicates, wrong dtypes, and outliers — and produces a human-readable report explaining every decision it made.

---

✨ Features

• 🧹 Automated Cleaning — handles duplicates, missing values, data types, and outliers
• ⚙️ Configurable — customize every threshold via CleaningConfig
• 📊 Explainable — detailed report of every action taken
• 🔍 Preview Mode — see what would be cleaned before applying
• 🛡️ Safe — input validation, memory limits, and error recovery
• 📈 Quality Metrics — before/after comparison of data quality
• 🧪 Well-Tested — comprehensive test suite

---

🚀 Quick Start

Installation

# From source
git clone https://github.com/cleanix/cleanix.git
cd cleanix
pip install -e ".[dev]"

Basic Usage

import pandas as pd
from cleanix import clean

df = pd.read_csv("your_dataset.csv")

cleaned_df, report = clean(df)

print(report.summary())
cleaned_df.to_csv("clean_data.csv", index=False)

Preview Changes First

from cleanix import suggest_cleaning

for s in suggest_cleaning(df):
    print(f"[{s['priority']}] {s['message']}")

Custom Configuration

from cleanix import clean, CleaningConfig

config = CleaningConfig(
    missing_drop_threshold=0.3,   # drop columns with >30% missing
    outlier_iqr_factor=2.0,       # more lenient outlier detection
    enable_duplicates=True,
    verbose=True,
)

cleaned_df, report = clean(df, config)

---

📖 API Reference

clean(df, config=None) → (DataFrame, Report)

Runs the full pipeline:

Stage	What it does
1. Duplicates	Drops exact duplicate rows
2. Missing	Imputes or drops columns with NaN
3. Data Types	Coerces object columns to better types
4. Outliers	Caps outliers with the IQR method

The original DataFrame is never mutated — a fresh copy is returned.

---

suggest_cleaning(df, config=None) → list[dict]

Preview what clean() would do, without modifying anything.

from cleanix import suggest_cleaning

for s in suggest_cleaning(df):
    print(s["priority"], s["message"])

---

profile_data(df) → dict[str, dict]

Per-column statistics: dtype, missing %, unique count, sample values.

from cleanix import profile_data

for col, info in profile_data(df).items():
    print(f"{col}: {info['missing_pct']:.1%} missing")

---

CleaningConfig

from cleanix import CleaningConfig

config = CleaningConfig(
    # Missing values
    missing_drop_threshold=0.50,         # drop if ≥50% missing

    # Type conversion
    numeric_conversion_threshold=0.80,   # convert if ≥80% numeric
    datetime_conversion_threshold=0.80,  # convert if ≥80% dates
    category_unique_ratio=0.05,          # convert if ≤5% unique
    category_min_rows=10,

    # Outliers
    outlier_iqr_factor=1.5,              # IQR multiplier
    outlier_min_rows=10,                 # skip if fewer rows

    # Pipeline toggles
    enable_duplicates=True,
    enable_missing=True,
    enable_datatypes=True,
    enable_outliers=True,

    # Safety
    max_memory_mb=None,                  # None = no limit
    verbose=False,
)

---

Report

Method	Returns	Description
.summary()	str	Human-readable, emoji-annotated summary
.to_dict()	dict	Plain Python dict of all actions
.to_json()	str	JSON-serialized report

---

🔧 Cleaning Strategy

Missing Values

Condition	Action
≥ 50% missing	Drop column
Numeric	Fill with median
Categorical	Fill with mode
100% missing	Drop column

Data Types

Condition	Action
≥ 80% values numeric	Convert to float64
≥ 80% values datetime	Convert to datetime64
Unique ratio ≤ 5%	Convert to category

Outliers

• IQR method: [Q1 − 1.5×IQR, Q3 + 1.5×IQR]
• Values outside fences are clipped (not dropped)
• Skipped on datasets with fewer than 10 rows

---

📊 Quality Metrics

from cleanix import clean, calculate_quality_metrics, format_quality_report

cleaned_df, report = clean(df)

metrics = calculate_quality_metrics(df, cleaned_df)
print(format_quality_report(metrics))

---

🛡️ Error Handling

from cleanix import clean, ValidationError, CleaningError

try:
    cleaned_df, report = clean(df)
except ValidationError as e:
    print(f"Bad input: {e}")
except CleaningError as e:
    print(f"Pipeline failed: {e}")

Exception hierarchy:

CleanixError
├── ValidationError    — bad input
├── ConfigurationError — bad config
├── CleaningError      — pipeline failure
└── MemoryLimitError   — exceeded memory limit

---

🧪 Running Tests

pip install -e ".[dev]"
pytest
pytest --cov=cleanix --cov-report=html   # with coverage

---

📋 Project Layout

cleanix/
├── cleanix/              # Main package
│   ├── __init__.py      # Public API
│   ├── cleaner.py       # clean() orchestrator
│   ├── config.py        # CleaningConfig
│   ├── exceptions.py    # Custom exceptions
│   ├── validation.py    # Input validation
│   ├── duplicates.py    # Duplicate removal
│   ├── missing.py       # Missing value handling
│   ├── datatypes.py     # Type coercion
│   ├── outliers.py      # Outlier capping
│   ├── profiler.py      # profile_data()
│   ├── suggester.py     # suggest_cleaning()
│   ├── report.py        # Report class
│   └── metrics.py       # Quality metrics
├── tests/               # Test suite
├── example_usage.py     # Usage examples
├── pyproject.toml
├── CONTRIBUTING.md
└── README.md

---

🤝 Contributing

See CONTRIBUTING.md for setup and guidelines.

---

📄 License

MIT — see LICENSE.