datapilot-polars 0.4.0

Your Data Science Copilot — Polars-native EDA, auto-cleaning, drift detection, and multi-provider AI insights

📖 DataPilot — Your Data Science Copilot

The only EDA library that is Polars-native, multi-provider AI-powered, and actually cleans your data.

DataPilot is an open-source Python library that automates Exploratory Data Analysis, detects data quality issues, cleans datasets, benchmarks performance, and generates intelligent AI recommendations — via Ollama (local), OpenAI, Google Gemini, Anthropic Claude, or Groq — all with minimal code.

---

🚀 Why DataPilot?

	ydata-profiling	sweetviz	dtale	DataPilot
Polars-native (10x faster)	❌	❌	❌	✅
Local AI (Ollama, private)	❌	❌	❌	✅
Cloud AI (OpenAI/Gemini/Claude/Groq)	❌	❌	❌	✅
Auto data cleaning	❌	❌	❌	✅
Train/test drift detection	✅	✅	❌	✅
Outlier detection	⚠️	❌	✅	✅
Smart column suggestions	❌	❌	❌	✅
Offline HTML dashboard	✅	✅	❌	✅
Regression diagnostics	❌	❌	❌	✅
ML model diagnostics	❌	❌	❌	✅

---

📑 Table of Contents

1. Installation
2. Quick Start
3. Module 1: Global Session Configuration
4. Module 2: Auto EDA Pipeline
5. Module 3: Smart Column Suggestions
6. Module 4: Dataset Analysis API
7. Module 5: Outlier Detection
8. Module 6: Auto Data Cleaning
9. Module 7: Train/Test Drift Detection
10. Module 8: Conversational AI (Ask AI)
11. Module 9: Visualization Engine
12. Module 10: Machine Learning Diagnostics
13. Module 11: Standalone HTML Dashboard
14. Module 12: Performance Benchmark
15. Module 13: AI Copilot Providers
16. Troubleshooting

---

Installation

From PyPI (Recommended)

To install the stable release:

pip install datapilot-polars

To install with optional cloud AI provider dependencies:

pip install "datapilot-polars[openai]"    # OpenAI support
pip install "datapilot-polars[gemini]"    # Google Gemini support
pip install "datapilot-polars[claude]"    # Anthropic Claude support
pip install "datapilot-polars[groq]"      # Groq support (free tier)
pip install "datapilot-polars[all-ai]"    # All cloud AI providers at once

From Source (Development)

# Clone and install in editable mode
git clone https://github.com/nx-manoj/DataPilot.git
cd DataPilot

uv venv && source .venv/bin/activate
uv pip install -e .

# For development / running tests
uv pip install -e .[dev]

---

Quick Start

import pandas as pd
import datapilot as dp

# 1. Configure AI provider once (Optional)
dp.configure(ai_provider="groq", api_key="gsk_...")

df = pd.read_csv("your_dataset.csv")

# 2. Full automated EDA with AI insights
dp.analyze(df, use_ai=True)

# 3. Get preprocessing suggestions with AI recommendations
suggestions = dp.suggest(df, use_ai=True)

# 4. Ask the AI conversational questions about the data
dp.ask_ai(df, "What are the most important preprocessing steps for this dataset?")

# 5. Generate plots with natural language prompts
dp.visualize_ai(df, "Show the relation between Age and Survived")

# 6. Auto-clean the dataset
clean_df, log = dp.auto_clean(df, use_ai=True)

# 7. Export a full offline HTML report
dp.dashboard(df, "report.html")

---

Module 1: Global Session Configuration

dp.configure(ai_provider="ollama", ai_model=None, api_key=None)

Set your credentials once at the beginning of your session. Subsequent calls containing use_ai=True will automatically fetch these credentials.

# Configure cloud AI (e.g., Groq)
dp.configure(ai_provider="groq", api_key="gsk_...")

# Subsequent calls don't need credentials repeated
dp.analyze(df, use_ai=True)
dp.suggest(df, use_ai=True)

---

Module 2: Auto EDA Pipeline

dp.analyze(df, use_ai=False, ai_provider=None, ai_model=None, api_key=None)

Runs all structural checks simultaneously and prints a clean console report.

# Standard rule-based analysis
dp.analyze(df)

# With configured AI copilot
dp.analyze(df, use_ai=True)

---

Module 3: Smart Column Suggestions

dp.suggest(df, use_ai=False, ai_provider=None, ai_model=None, api_key=None)

Analyses every column and returns actionable, rule-based preprocessing recommendations. Enables optional use_ai=True to append expert AI comments.

suggestions = dp.suggest(df, use_ai=True)

---

Module 4: Dataset Analysis API

dp.summary(df)

Returns a high-level overview dict:

meta = dp.summary(df)
# {'rows': 891, 'columns': 12, 'memory_usage_mb': 0.08, 'engine_detected': 'pandas', ...}

dp.missing(df)

Returns sorted DataFrame showing column null counts and percentages.

dp.duplicates(df)

Checks for exact duplicate rows across all CPU cores.

dp.correlation(df, threshold=0.6)

Calculates the Pearson correlation matrix for all numeric columns, flagging strong pairs.

---

Module 5: Outlier Detection

dp.outliers(df, method="both", z_threshold=3.0, iqr_multiplier=1.5, use_ai=False, ...)

Detects outliers across numeric columns using IQR fencing and/or Z-score. Set use_ai=True to receive AI recommendations on how to handle them.

result = dp.outliers(df, use_ai=True)

---

Module 6: Auto Data Cleaning

dp.auto_clean(df, drop_null_threshold=0.6, impute_strategy="auto", drop_id_columns=True, drop_constant_columns=True, use_ai=False, ...)

Automatically cleans the dataset and logs changes. Enabling use_ai=True appends a conversational explanation of why the actions improve model quality.

clean_df, change_log = dp.auto_clean(df, use_ai=True)

---

Module 7: Train/Test Drift Detection

dp.compare(df_train, df_test, threshold=0.1, use_ai=False, ...)

Detects distribution shift between training and test datasets. Uses Jensen-Shannon divergence for categoricals. Enabling use_ai=True yields AI-suggested mitigation strategies.

flags = dp.compare(df_train, df_test, use_ai=True)

---

Module 8: Conversational AI (Ask AI)

dp.ask_ai(df, question, ai_provider=None, ai_model=None, api_key=None)

Ask free-form natural-language questions about your dataset. Only statistical metadata is transmitted to the AI — never raw rows.

dp.ask_ai(df, "Which features carry the most risk of data leakage?")
dp.ask_ai(df, "Should I log-transform Fare or normalise Age first?")

---

Module 9: Visualization Engine

Includes publication-ready, dark-themed plots (#0f172a slate background) with automatic statistical overlays.

dp.hist(df, column, bins="auto", hue=None, color="#3b82f6")

Histogram with automatic KDE overlay, plus mean and median lines.

dp.box(df, column, group_by=None, orient="v")

Box plot with median highlights and automatic IQR annotation.

dp.heatmap(df)

Lower-triangle Pearson correlation matrix heatmap.

dp.scatter(df, x, y, hue=None, trendline=True)

Scatter plot with optional OLS regression trendline.

dp.violin(df, column, group_by=None)

Violin plot combining box plot and KDE for rich distribution insights.

dp.visualize_ai(df, prompt, ai_provider=None, ai_model=None, api_key=None)

Ask the AI to choose and draw the right chart from a plain-English prompt.

dp.visualize_ai(df, "Show the relation between Age and Survived")
dp.visualize_ai(df, "Distribution of Fare for each passenger class")
dp.visualize_ai(df, "Correlation heatmap of numeric columns")

---

Module 10: Machine Learning Diagnostics

dp.classification_report(y_true, y_pred, average="auto")

Calculates binary or multi-class metrics safely.

dp.regression_report(y_true, y_pred)

Calculates MAE, MSE, RMSE, R², MAPE, and Max Error.

dp.diagnose(train_score, test_score, metric_name="Accuracy")

Evaluates train/test performance gaps to diagnose overfitting or underfitting.

---

Module 11: Standalone HTML Dashboard

dp.dashboard(df, output_path="datapilot_report.html")

Generates a complete, offline-ready HTML dashboard report containing metrics, datatype profiles, missing value charts, and correlation heatmap matrix.

---

Module 12: Performance Benchmark

dp.benchmark(df)

Benchmarks DataPilot (Polars core) operations against equivalent Pandas operations.

---

Module 13: AI Copilot Providers

DataPilot uses a Metadata-Only AI Pattern — raw data rows are never transmitted. Only statistical summaries are sent.

Provider	Type	Default Model	Requires
ollama	🔒 Local / Private	llama3	Ollama daemon running locally
openai	☁️ Cloud	gpt-4o-mini	pip install datapilot-polars[openai] + API key
gemini	☁️ Cloud	gemini-1.5-flash	pip install datapilot-polars[gemini] + API key
claude	☁️ Cloud	claude-3-haiku-20240307	pip install datapilot-polars[claude] + API key
groq	☁️ Cloud (free tier)	llama3-70b-8192	pip install datapilot-polars[groq] + API key

---

Troubleshooting

ModuleNotFoundError after editing files

uv pip install -e . --force-reinstall

AI: Ollama Connection Refused

Ensure the local daemon is active:

ollama serve

Running Tests

pytest -v
pytest --cov=datapilot