SmartClean-AI

Automated Structured-Data Cleaning and Reporting in Python

SmartClean-AI is a Python library designed to help prepare structured datasets for analysis by addressing common data-quality issues and providing access to cleaning reports and dataset profiles.

Project links

• PyPI: https://pypi.org/project/smartclean-ai/
• GitHub: https://github.com/yashtalati10/SmartClean
• Issues and bug reports: https://github.com/yashtalati10/SmartClean/issues

---

Table of Contents

1. Overview
2. Features
3. Installation
4. Requirements
5. Quick Start
6. Complete Output Boilerplate
7. Understanding the Output Files
8. Working with CleaningResult
9. Input Data
10. Example Project Structure
11. Troubleshooting
12. Contributing
13. License

---

1. Overview

Real-world datasets may contain missing values, duplicate records, inconsistent data, and outliers. These issues can complicate exploratory data analysis (EDA) and downstream machine-learning workflows.

SmartClean-AI provides a Python interface for structured-data cleaning and exposes cleaning results, reports, and dataset profiles.

The library is intended to help users prepare data more efficiently while keeping the cleaned dataset and report files available for inspection.

2. Features

• Missing-value handling: Handle missing values through the implemented cleaning pipeline.
• Duplicate handling: Support duplicate-record handling where configured in the pipeline.
• Outlier handling: Support outlier processing through the implemented handler and selected method.
• Dataset profiles: Access before-cleaning and after-cleaning profiles through CleaningResult.
• Cleaning summaries: Display a formatted summary using display_report().
• CSV export: Save the cleaned DataFrame using save_csv().
• JSON reports: Save structured cleaning information using save_report().
• TXT reports: Export a readable report using save_report().
• Custom reporting: Use the result object to build additional reports and visualizations.

The exact cleaning behavior depends on the installed version and the pipeline implementation.

3. Installation

Install the package from PyPI:

python -m pip install smartclean-ai

Check the installed version:

python -m pip show smartclean-ai

Import the public API:

from smartclean import Cleaner, CleaningResult

print("SmartClean-AI imports successfully.")

4. Requirements

The project metadata specifies these dependencies:

• Python 3.10 or later
• NumPy
• pandas
• Matplotlib
• scikit-learn

Install or upgrade the dependencies through pip if needed:

python -m pip install --upgrade numpy pandas matplotlib scikit-learn

5. Quick Start

The following example illustrates the intended workflow. Confirm the constructor and cleaning method against the installed smartclean/cleaner.py implementation before running it.

from smartclean import Cleaner

# Path to your input dataset
input_file = "data.csv"

# Initialize the cleaner
cleaner = Cleaner(input_file)

# Run the cleaning workflow
result = cleaner.clean()

# Display the cleaning report
result.display_report()

# Save the cleaned dataset
result.save_csv("cleaned_data.csv")

# Save JSON and TXT reports
result.save_report("cleaning_report.json", format="json")
result.save_report("cleaning_report.txt", format="txt")

print("Cleaning and export workflow completed.")

If your installed version uses a different method to start the cleaning workflow, replace cleaner.clean() with the actual method implemented by Cleaner.

6. Complete Output Boilerplate

This example shows how a user can organize a single run to produce four deliverables:

1. Cleaned CSV
2. JSON cleaning report
3. TXT cleaning report
4. Dashboard PNG

The CSV and reports use the CleaningResult methods shown in the project source. The dashboard section is an example integration and requires a compatible dashboard function.

Save the following as run_smartclean.py after verifying the cleaning call and dashboard function signature.

from pathlib import Path
import json
import pandas as pd
import matplotlib.pyplot as plt

from smartclean import Cleaner

# --------------------------------------------------
# CONFIGURATION
# --------------------------------------------------

INPUT_FILE = Path("data.csv")
OUTPUT_DIR = Path("smartclean_output")

OUTPUT_DIR.mkdir(parents=True, exist_ok=True)

CSV_OUTPUT = OUTPUT_DIR / "cleaned_data.csv"
JSON_OUTPUT = OUTPUT_DIR / "cleaning_report.json"
TXT_OUTPUT = OUTPUT_DIR / "cleaning_report.txt"
DASHBOARD_OUTPUT = OUTPUT_DIR / "cleaning_dashboard.png"


# --------------------------------------------------
# RUN CLEANING
# --------------------------------------------------

def main():
    if not INPUT_FILE.exists():
        raise FileNotFoundError(
            f"Input dataset was not found: {INPUT_FILE.resolve()}"
        )

    print(f"Input dataset: {INPUT_FILE.resolve()}")

    # Verify this call against the installed Cleaner API.
    cleaner = Cleaner(str(INPUT_FILE))
    result = cleaner.clean()

    # --------------------------------------------------
    # 1. DISPLAY REPORT
    # --------------------------------------------------

    result.display_report()

    # --------------------------------------------------
    # 2. EXPORT CLEANED CSV
    # --------------------------------------------------

    result.save_csv(str(CSV_OUTPUT))

    # --------------------------------------------------
    # 3. EXPORT JSON REPORT
    # --------------------------------------------------

    result.save_report(str(JSON_OUTPUT), format="json")

    # --------------------------------------------------
    # 4. EXPORT TXT REPORT
    # --------------------------------------------------

    result.save_report(str(TXT_OUTPUT), format="txt")

    # --------------------------------------------------
    # 5. GENERATE A BASIC DATASET DASHBOARD
    # --------------------------------------------------

    cleaned_df = result.cleaned_df
    before_df = pd.read_csv(INPUT_FILE)

    missing_before = int(before_df.isna().sum().sum())
    missing_after = int(cleaned_df.isna().sum().sum())

    rows_before = len(before_df)
    rows_after = len(cleaned_df)

    columns_before = len(before_df.columns)
    columns_after = len(cleaned_df.columns)

    fig, axes = plt.subplots(1, 2, figsize=(11, 5))

    # Dataset dimensions
    axes[0].bar(
        ["Before", "After"],
        [rows_before, rows_after],
        label="Rows"
    )
    axes[0].set_title("Dataset Row Count")
    axes[0].set_ylabel("Number of rows")

    # Missing-value counts
    axes[1].bar(
        ["Before", "After"],
        [missing_before, missing_after],
        label="Missing values"
    )
    axes[1].set_title("Missing Values")
    axes[1].set_ylabel("Number of missing values")

    fig.suptitle("SmartClean-AI Cleaning Dashboard")
    fig.tight_layout()
    fig.savefig(DASHBOARD_OUTPUT, dpi=160, bbox_inches="tight")
    plt.close(fig)

    # --------------------------------------------------
    # COMPLETION SUMMARY
    # --------------------------------------------------

    print("\nAll requested output files have been generated:")
    print(f"Cleaned CSV:    {CSV_OUTPUT.resolve()}")
    print(f"JSON report:    {JSON_OUTPUT.resolve()}")
    print(f"TXT report:     {TXT_OUTPUT.resolve()}")
    print(f"Dashboard PNG:  {DASHBOARD_OUTPUT.resolve()}")


if __name__ == "__main__":
    main()

Important notes about the boilerplate

• The example assumes Cleaner(input_path) and cleaner.clean() are supported. Verify those two calls before publishing this as a guaranteed runnable example.
• The dashboard above is generated by the example script using Matplotlib. It is not represented as a built-in CleaningResult dashboard method.
• This example reads the input with pandas.read_csv(). For Excel input, the dashboard's input-loading line must be adapted to the supported Excel format.
• The JSON and TXT report methods serialize the report and available profiles; they do not automatically include arbitrary dashboard metrics unless those metrics are part of the stored report.
• The unused json import can be removed from this example; the report export is handled by result.save_report().

7. Understanding the Output Files

After a successful run, the output directory should look like this:

smartclean_output/
├── cleaned_data.csv
├── cleaning_report.json
├── cleaning_report.txt
└── cleaning_dashboard.png

File	Purpose
cleaned_data.csv	Contains the cleaned dataset.
cleaning_report.json	Stores structured report and profile information.
cleaning_report.txt	Stores a human-readable report.
cleaning_dashboard.png	Visualizes selected before-and-after dataset statistics.

The dashboard values should be interpreted alongside the report. Fewer missing values or fewer rows do not automatically mean every dataset is better; review the transformations before using the results.

8. Working with CleaningResult

The public package exports CleaningResult, which provides access to the cleaned dataset, profiles, and cleaning report.

Access the cleaned DataFrame

cleaned_df = result.cleaned_df

print(cleaned_df.head())
print(cleaned_df.shape)

Display the report

result.display_report()

Save the cleaned dataset

result.save_csv("output/cleaned_data.csv")

Save a JSON report

result.save_report("output/report.json", format="json")

Save a TXT report

result.save_report("output/report.txt", format="txt")

Inspect the report and profiles

print(result.report)
print(result.before_profile)
print(result.after_profile)

These objects can be used to build custom summaries and visualizations. The exact keys and nested structure depend on the reports returned by the implemented cleaning handlers.

9. Input Data

Prepare a CSV or Excel file supported by the installed data loader and pass its path to the cleaning workflow.

Example CSV:

Name,Age,City,Salary
Aarav,25,Pune,45000
Meera,,Mumbai,52000
Kabir,32,Pune,
Aarav,25,Pune,45000

This small dataset illustrates missing values and a repeated record. It is an illustrative input, not a guarantee of how every cleaning setting will transform the rows.

For Excel input, use an extension and loading method supported by the version you have installed.

10. Example Project Structure

my-data-project/
├── data.csv
├── run_smartclean.py
└── smartclean_output/
    ├── cleaned_data.csv
    ├── cleaning_report.json
    ├── cleaning_report.txt
    └── cleaning_dashboard.png

Keep the original input file unchanged so that you can compare the original dataset with the output.

11. Troubleshooting

Import error

ModuleNotFoundError: No module named 'smartclean'

Install the package in the active Python environment:

python -m pip install smartclean-ai

Input file not found

Check the path and filename before running the script. Relative paths are resolved from the script's current working directory.

Report method or attribute error

Confirm the installed package version and inspect the Cleaner and CleaningResult APIs. Do not assume a method exists unless it is implemented in that version.

Excel loading error

Check whether the file extension is supported and whether the necessary Excel-reading dependency is installed.

Dashboard generation error

Ensure Matplotlib is installed and that the dashboard code uses numeric statistics available in the selected dataset.

12. Contributing

Contributions, bug reports, and feature suggestions are welcome.

1. Fork the repository.
2. Create a branch for your change.
3. Test your changes locally.
4. Submit a pull request with a description of the change.

Repository: https://github.com/yashtalati10/SmartClean

13. License

This project declares the MIT license in its package metadata. Refer to the repository's license file for the complete license terms.