SmartClean-AI

An intelligent Python library for automated data cleaning with profiling, preprocessing, and detailed reporting.

SmartClean-AI simplifies data preparation by identifying and handling common data quality issues, including missing values, duplicate records, and outliers. It generates profiling information before and after cleaning to help users understand changes to their datasets.

• PyPI: https://pypi.org/project/smartclean-ai/
• GitHub: https://github.com/yashtalati10/SmartClean
• Current documented version: 1.1.1

Features

• Automated missing-value detection and handling
• Duplicate row detection and removal
• Outlier detection using the IQR method
• Automatic data profiling
• Before-and-after cleaning reports
• Configurable cleaning pipeline
• Modular and extensible architecture
• Simple Python API
• Open-source and developer-friendly

Installation

Install SmartClean-AI from PyPI:

python -m pip install --upgrade smartclean-ai

To install from the source repository instead:

git clone https://github.com/yashtalati10/SmartClean.git
cd SmartClean
python -m pip install .

Quick Start

1. Prepare your dataset

Place your CSV file in the same directory as your Python script. This example uses participants.csv.

2. Run SmartClean

import pandas as pd
from smartclean import Cleaner

# Load the dataset
df = pd.read_csv("participants.csv")

# Initialize SmartClean
cleaner = Cleaner()

# Run the cleaning pipeline
result = cleaner.clean(df)

# Access the cleaned DataFrame
cleaned_df = result.cleaned_df

# Display the first five rows
print(cleaned_df.head())

# Save the cleaned dataset
cleaned_df.to_csv("cleaned_data.csv", index=False)
print("Cleaned data saved to cleaned_data.csv")

# Compare dataset dimensions
print("Original shape:", df.shape)
print("Cleaned shape:", cleaned_df.shape)

# Display the cleaning report
print(result.report)

The cleaned CSV is saved as cleaned_data.csv in the current working directory.

Example Output

The following is an illustrative example of the kind of messages a cleaning run may produce. Exact values and messages depend on the input dataset and the library version.

========== Missing Value Cleaning ==========

[OK] Column: Age       | Missing: 177 | Strategy: mean
[OK] Column: Cabin     | Missing: 687 | Strategy: most_frequent
[OK] Column: Embarked  | Missing: 2   | Strategy: most_frequent

========== Duplicate Cleaning ==========

[OK] No duplicate rows found.

========== Outlier Cleaning ==========

[OK] Outlier Values Capped: 441

These numbers are illustrative, not guaranteed results for every dataset.

Cleaning Pipeline

Input Dataset
      |
      v
Data Profiling
      |
      v
Missing Value Handling
      |
      v
Duplicate Removal
      |
      v
Outlier Detection and Handling
      |
      v
Updated Data Profiling
      |
      v
Cleaning Report
      |
      v
Cleaned Dataset

Current Cleaning Capabilities

Missing Values

Depending on the configured cleaning behavior, SmartClean supports:

• Mean imputation
• Median imputation
• Most-frequent-value imputation
• Dropping columns with excessive missing values

Duplicate Records

• Detect duplicate rows
• Remove duplicate records

Outliers

• IQR-based outlier detection
• Outlier capping
• Outlier removal

Accessing Cleaning Results

The Cleaner.clean() method returns a CleaningResult object.

result = cleaner.clean(df)

# Cleaned DataFrame
cleaned_df = result.cleaned_df

# Profile before cleaning
before_profile = result.before_profile

# Profile after cleaning
after_profile = result.after_profile

# Cleaning report
report = result.report

Save the cleaned DataFrame with:

result.cleaned_df.to_csv("cleaned_data.csv", index=False)

Project Structure

The following is the documented project structure. Exact files may vary by repository version.

SmartClean/
├── smartclean/
│   ├── cleaning/
│   ├── config/
│   ├── core/
│   ├── handlers/
│   ├── utils/
│   ├── cleaner.py
│   ├── result.py
│   └── __init__.py
├── example/
│   └── demo.py
├── data/
├── reports/
├── README.md
├── pyproject.toml
└── requirements.txt

Roadmap

Version 1.0 — Core Cleaning Features

• Data profiling
• Missing-value handling
• Duplicate removal
• Outlier detection
• Cleaning reports

Further roadmap items will be documented as they are implemented and verified.

Dependencies

SmartClean-AI declares the following runtime dependencies:

• NumPy
• pandas
• Matplotlib
• scikit-learn

These dependencies are installed automatically when you install the package from PyPI.

Contributing

Contributions, bug reports, and feature suggestions are welcome.

• Repository: https://github.com/yashtalati10/SmartClean
• Issue tracker: https://github.com/yashtalati10/SmartClean/issues

When reporting a problem, include your Python version, SmartClean-AI version, a minimal example, and the full error message.

License

SmartClean-AI is distributed under the MIT License.