SmartClean-AI

Automated Data Cleaning, Profiling, and Reporting for Python

SmartClean-AI is a Python library designed to simplify structured-data cleaning and exploratory data analysis (EDA). It helps identify and handle common data-quality problems in CSV and Excel datasets, including missing values, columns with excessive missing data, and outliers.

The library provides a configurable cleaning workflow, a cleaned dataset, and reports that help users understand the changes made during preprocessing.

• PyPI: https://pypi.org/project/smartclean-ai/
• GitHub: https://github.com/yashtalati10/SmartClean
• Package name: smartclean-ai
• Python import: smartclean
• Documented version: 1.2.0 — update this to match the latest published release.

---

Table of Contents

• Overview
• Features
• How It Works
• Installation
• Quick Start
• Command-Line Workflow
• Configuration
• Output Files
• Dashboard Preview
• Example JSON Report
• Example TXT Report
• Using the Python API
• Project Structure
• Dependencies
• Limitations and Notes
• Contributing
• License

---

Overview

Data preparation is an important step in data analysis and machine learning. Real-world datasets can contain missing values, redundant records, inconsistent data, and extreme values that affect downstream analysis.

SmartClean-AI aims to make this process easier by providing reusable cleaning components and reporting capabilities.

The project focuses on:

• Detecting and handling missing values.
• Identifying columns with excessive missing data.
• Applying outlier-handling techniques.
• Saving cleaned datasets.
• Producing structured JSON and human-readable TXT reports.
• Generating a cleaning dashboard for reviewing results.

Features

Data Cleaning

• Automated missing-value handling.
• Configurable threshold for dropping columns with excessive missing data.
• Outlier handling with configurable methods supported by the implementation.
• CSV and Excel input support through the data-loading component.
• Reusable cleaning components.

Reporting and Analysis

• Before-and-after dataset dimensions.
• Missing-value statistics.
• Number of columns dropped.
• Outlier-handling statistics.
• Detailed dropped-column information in reports.
• JSON report for programmatic access.
• TXT report for human-readable results.
• Dashboard image for visual inspection.
• Concise terminal summary.

Developer Experience

• Python-based workflow.
• Modular package design.
• Configurable cleaning parameters.
• Suitable for data preparation and preprocessing workflows.

Note: Duplicate removal, specific imputation strategies, and individual outlier-removal modes should only be advertised as supported options when they are implemented and verified in the published package version.

---

How It Works

The structured-data cleaning workflow follows these general stages:

             Input Dataset
                   |
                   v
             Data Loading
                   |
                   v
          Initial Data Inspection
                   |
                   v
          Missing-Value Handling
                   |
                   v
        Excess-Missing Column Check
                   |
                   v
           Outlier Handling
                   |
                   v
         Save Cleaned Dataset
                   |
          +--------+--------+
          |        |        |
          v        v        v
        JSON      TXT    Dashboard
        Report   Report    Image
                   |
                   v
           Cleaning Summary

The precise behavior depends on the selected configuration and the cleaning components available in the installed version.

---

Installation

Install the published package from PyPI:

python -m pip install --upgrade smartclean-ai

Verify that the package can be imported:

python -c "import smartclean; print('SmartClean-AI import successful')"

Install from GitHub

git clone https://github.com/yashtalati10/SmartClean.git
cd SmartClean
python -m pip install .

For development work, install the project in editable mode if the repository's build configuration supports it:

python -m pip install -e .

---

Quick Start

The following example demonstrates the intended workflow for users who already have access to a compatible Cleaner API.

import pandas as pd
from smartclean import Cleaner

# Load a dataset.
df = pd.read_csv("enter_file_path.csv")

# Initialize the cleaner.
cleaner = Cleaner()

# Run the cleaning workflow.
result = cleaner.clean(df)

# Access the cleaned DataFrame.
cleaned_df = result.cleaned_df

# Save the result.
cleaned_df.to_csv("cleaned_data.csv", index=False)

# Compare the dataset dimensions.
print("Original shape:", df.shape)
print("Cleaned shape:", cleaned_df.shape)

# Inspect the returned report.
print(result.report)

API compatibility: This example follows the API documented in the previous README. Confirm that the published version you install exposes Cleaner() and cleaner.clean(df) before using it. The command-line workflow below is documented separately.

---

Command-Line Workflow

The project also contains a module-based cleaning workflow.

1. Prepare the input dataset

For the default project configuration, place the input CSV at:

data/
└── raw/
    └── data.csv

2. Run the cleaning module

From the project root, execute:

python -m smartclean.main

The workflow uses the configured input path, applies missing-value handling and outlier handling, and writes its outputs to the cleaned-data directory.

3. Review the results

The default output directory discussed for this workflow is:

data/
└── cleaned/
    ├── cleaned_data.csv
    ├── cleaning_report.json
    ├── cleaning_report.txt
    └── cleaning_dashboard.png

Your actual paths may differ if you configure the input or output directories.

---

Configuration

The module-based workflow has parameters for the input file, outlier-handling method, and missing-value threshold.

The following values reflect the discussed main() function signature:

Parameter	Purpose	Example
file_path	Optional input dataset path	data/raw/data.csv
outlier_method	Outlier-handling method	"cap"
drop_threshold	Missing-value percentage threshold for dropping columns	90

For example, the function has been structured to accept:

main(
    file_path="data/raw/data.csv",
    outlier_method="cap",
    drop_threshold=90,
)

This example assumes the module's main() function is imported and that the specified path exists.

Missing-value threshold

A threshold of 90 means that columns exceeding the configured missing-value percentage are candidates for removal, subject to the implementation's comparison rule.

For example, a column with 96% missing values exceeds a 90% threshold.

Outlier handling

The example configuration uses "cap" for outlier handling. Use only method names accepted by the installed OutlierHandler implementation.

---

Output Files

The module-based workflow is designed to generate the following artifacts.

File	Purpose
cleaned_data.csv	Dataset after the cleaning workflow
cleaning_report.json	Structured report suitable for scripts and further analysis
cleaning_report.txt	Human-readable summary with detailed dropped-column information
cleaning_dashboard.png	Visual summary of cleaning results

Terminal Summary

The terminal output is intended to show concise statistics rather than print every dropped column.

A representative summary might look like this:

========== SmartClean-AI Summary ==========

Original rows:              11,933
Cleaned rows:               11,933
Original columns:           1,112
Cleaned columns:             930

Missing values before:  7,853,567
Missing values filled:  5,740,609
Missing values remaining:       0

Columns dropped:                182
Outlier values capped:     1,741,216

Cleaning completed.
Reports and dashboard saved.

Illustrative output: The figures above come from a previously discussed run and should not be interpreted as guaranteed results. Actual values depend on the input dataset and configuration.

The detailed dropped-column list belongs in the generated reports rather than the concise terminal summary.

---

Dashboard Preview

SmartClean-AI's workflow is intended to generate a dashboard image at:

data/cleaned/cleaning_dashboard.png

To display it in this README, add a copy of the actual generated image to the repository. A convenient location is:

docs/images/cleaning_dashboard.png

Then use this Markdown:

![SmartClean-AI Cleaning Dashboard](docs/images/cleaning_dashboard.png)

The rendered image should appear here after you commit the image file to GitHub.

Important: No real dashboard image was supplied with this README content, so the image reference is a placeholder until you add your actual generated dashboard.

---

Example JSON Report

The JSON report is intended for structured access to dataset statistics, cleaning results, and detailed column information.

The following is an illustrative schema example, not a guaranteed copy of the generated report. Keep the actual field names from your build_full_report() implementation when documenting a real report.

{
  "original_shape": [11933, 1112],
  "cleaned_shape": [11933, 930],
  "missing_report": {
    "missing_values_before": 7853567,
    "missing_values_filled": 5740609,
    "missing_values_remaining": 0,
    "columns_dropped": 182,
    "drop_threshold": 90.0
  },
  "dropped_columns": [
    {
      "column": "example_column",
      "missing_percentage": 96.84
    }
  ],
  "outlier_report": {
    "outlier_values_capped": 1741216
  },
  "outputs": {
    "cleaned_dataset": "cleaned_data.csv",
    "text_report": "cleaning_report.txt",
    "dashboard": "cleaning_dashboard.png"
  }
}

The column name shown above is a placeholder. In the real report, the dropped-column section should contain the actual column names and statistics produced by the cleaning run.

Reading the JSON report

import json
from pathlib import Path

report_path = Path("data/cleaned/cleaning_report.json")

with report_path.open("r", encoding="utf-8") as file:
    report = json.load(file)

print(json.dumps(report, indent=2))

The report can be used for automated validation, experiment records, or integration into other data-processing workflows.

---

Example TXT Report

The text report is intended to provide a readable summary of the cleaning process, including the detailed dropped-column list.

Illustrative example:

========================================
       SMARTCLEAN-AI CLEANING REPORT
========================================

Original dataset shape: (11933, 1112)
Cleaned dataset shape:  (11933, 930)

MISSING-VALUE HANDLING
----------------------
Missing values before: 7,853,567
Missing values filled: 5,740,609
Missing values remaining: 0
Columns dropped: 182
Drop threshold: 90.00%

DROPPED COLUMNS
---------------
- example_column: 96.84% missing
- another_example_column: 94.20% missing

OUTLIER HANDLING
----------------
Outlier values capped: 1,741,216

OUTPUT FILES
------------
cleaned_data.csv
cleaning_report.json
cleaning_report.txt
cleaning_dashboard.png

========================================

The example column names are placeholders, and the report layout may vary from the actual generated TXT file. A real report should preserve the full list of columns dropped during cleaning.

---

Using the Python API

The previous documented API exposes the cleaned DataFrame and report information through the returned result object.

result = cleaner.clean(df)

# Cleaned dataset
cleaned_df = result.cleaned_df

# Dataset profile before cleaning
before_profile = result.before_profile

# Dataset profile after cleaning
after_profile = result.after_profile

# Cleaning report
report = result.report

Save the cleaned data:

result.cleaned_df.to_csv("cleaned_data.csv", index=False)

The names and availability of these result attributes should be checked against the installed release. Refer to the repository source code for the authoritative API.

---

Project Structure

The following is a conceptual structure for the documented workflow. It is not intended to guarantee that every listed directory exists in every repository version.

SmartClean/
├── smartclean/
│   ├── __init__.py
│   ├── main.py
│   ├── cleaner.py
│   └── ...
├── data/
│   ├── raw/
│   └── cleaned/
├── docs/
│   └── images/
│       └── cleaning_dashboard.png
├── examples/
├── README.md
├── pyproject.toml
└── requirements.txt

The package's actual structure should be used when documenting specific modules or classes.

---

Dependencies

The previously documented runtime dependencies include:

• NumPy
• pandas
• Matplotlib
• scikit-learn

The package may use additional dependencies depending on the modules included in the release.

Install SmartClean-AI through PyPI to allow its declared runtime dependencies to be installed automatically. The authoritative dependency list is the package's published metadata and pyproject.toml.

---

Limitations and Notes

• Cleaning outcomes depend on the input dataset, configuration, and selected methods.
• Dropping a column with excessive missing values can remove useful information; review the report before using the cleaned dataset for analysis.
• Outlier capping changes extreme values and may not be appropriate for every dataset.
• A dataset with zero missing values after cleaning is not necessarily free of all data-quality problems.
• Review the generated reports before using the cleaned output in production or research.
• Examples in this README are illustrative unless explicitly identified as verified output.

---

Contributing

Contributions, bug reports, and feature suggestions are welcome.

• Repository: https://github.com/yashtalati10/SmartClean
• Issue tracker: https://github.com/yashtalati10/SmartClean/issues

When reporting a bug, include:

1. SmartClean-AI version.
2. Python version.
3. Operating system.
4. A minimal reproducible example.
5. The full error message, with sensitive data removed.

For code contributions, test the affected cleaning workflow and update the documentation when behavior or public APIs change.

---

License

The previous README identifies SmartClean-AI as distributed under the MIT License. Confirm that the repository contains the corresponding LICENSE file before publishing this statement.

---

Links

• PyPI package: https://pypi.org/project/smartclean-ai/
• GitHub repository: https://github.com/yashtalati10/SmartClean
• GitHub issues: https://github.com/yashtalati10/SmartClean/issues

SmartClean-AI — making structured-data cleaning and reporting easier for Python developers.