AIDataPilot 🚀

AIDataPilot is an intelligent, automated data cleaning and profiling library designed to transform messy datasets into analysis-ready data with minimal human intervention.

Key Features

• 🧠 Intelligent Decision Engine: Automatically determines the best cleaning strategies based on dataset characteristics.
• 🧹 Structural Cleaning: Standardizes column names, handles date formats (DD-MM-YYYY), and removes unnecessary white spaces.
• 💎 Duplicate Handling: Identifies and resolves exact and near-duplicates using normalization techniques.
• 📉 Outlier Management: Detects and caps outliers using IQR (Interquartile Range) to preserve data distribution without losing rows.
• ⚙️ Type Conversion: Automatically promotes columns to their most appropriate types (numeric, datetime, boolean, etc.).
• ❓ Missing Value Handling: Implements smart defaults and handles null values consistently across different data types.
• 📊 Quality Profiling: Generates comprehensive reports on data quality, structure, and the actions taken during cleaning.

Installation

pip install .

Note: Requires Python 3.8+ and dependencies: pandas, numpy, scikit-learn, scipy.

Quick Start

🚀 Full Intelligence Engine (Auto-Pilot)

Run the full suite (Profile → Validate → Clean → ML → Text → Report) in one command:

import aidatapilot

# Ingest any messy dataset
results = aidatapilot.auto_pilot("messy_data.csv", output_dir="results")

# results contains:
# - .data: The cleaned DataFrame
# - .report: Executive summary and quality scores

🧩 Individual Pipelines

You can also run specific pipelines independently:

• auto_clean(): Deep semantic cleaning and outlier handling.
• auto_validate(): Diagnostic data health reporting.
• auto_ml(): Feature engineering and model recommendations.
• auto_text(): Natural language narrative generation.
• auto_profile(): Detailed statistical distribution profiling.

Demos

Check the examples/ directory for standalone scripts demonstrating each pipeline:

• demo_clean.py, demo_ml.py, demo_report.py, etc.

Project Structure

• aidatapilot/core/: Core logic and shared context.
• aidatapilot/features/: Modular cleaning features (missing values, outliers, duplicates, etc.).
• aidatapilot/pipelines/: Orchestration layers for automated workflows.
• aidatapilot/utils/: Internal helper functions.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.