datapilot-polars 0.3.0

Your Data Science Copilot — Polars-native EDA, auto-cleaning, drift detection, and multi-provider AI insights

📖 DataPilot — Your Data Science Copilot

The only EDA library that is Polars-native, multi-provider AI-powered, and actually cleans your data.

DataPilot is an open-source Python library that automates Exploratory Data Analysis, detects data quality issues, cleans datasets, benchmarks performance, and generates intelligent AI recommendations — via Ollama (local), OpenAI, Google Gemini, Anthropic Claude, or Groq — all with minimal code.

---

🚀 Why DataPilot?

	ydata-profiling	sweetviz	dtale	DataPilot
Polars-native (10x faster)	❌	❌	❌	✅
Local AI (Ollama, private)	❌	❌	❌	✅
Cloud AI (OpenAI/Gemini/Claude/Groq)	❌	❌	❌	✅
Auto data cleaning	❌	❌	❌	✅
Train/test drift detection	✅	✅	❌	✅
Outlier detection	⚠️	❌	✅	✅
Smart column suggestions	❌	❌	❌	✅
Offline HTML dashboard	✅	✅	❌	✅
Regression diagnostics	❌	❌	❌	✅
ML model diagnostics	❌	❌	❌	✅

---

📑 Table of Contents

1. Installation
2. Quick Start
3. Module 1: Auto EDA Pipeline
4. Module 2: Smart Column Suggestions
5. Module 3: Dataset Analysis API
6. Module 4: Outlier Detection
7. Module 5: Auto Data Cleaning
8. Module 6: Train/Test Drift Detection
9. Module 7: Visualization Engine
10. Module 8: Machine Learning Diagnostics
11. Module 9: Standalone HTML Dashboard
12. Module 10: Performance Benchmark
13. Module 11: AI Copilot (5 Providers)
14. Installation & Setup
15. Troubleshooting

---

Installation

# Clone and install in editable mode
git clone https://github.com/yourusername/datapilot.git
cd datapilot

uv venv && source .venv/bin/activate
uv pip install -e .

# For development / running tests
uv pip install -e .[dev]

System requirement: Python 3.9+. Ollama is only required if you use use_ai=True.

---

Quick Start

import pandas as pd
import datapilot as dp

df = pd.read_csv("your_dataset.csv")

# Full automated EDA in one line
dp.analyze(df)

# Smart suggestions on what to fix
dp.suggest(df)

# Auto-clean the dataset
clean_df, log = dp.auto_clean(df)

# Export a full HTML report
dp.dashboard(df, "report.html")

---

Module 1: Auto EDA Pipeline

dp.analyze(df, use_ai=False, ai_model="llama3")

The primary entry point. Runs all structural checks simultaneously and prints a clean console report.

# Standard rule-based analysis
dp.analyze(df)

# With local AI copilot (requires Ollama)
dp.analyze(df, use_ai=True, ai_model="llama3")

Console Output:

==================================================
         DATAPILOT AUTOMATED EDA REPORT         
==================================================
📊 Dataset Shape: 891 rows × 12 columns
💾 Memory Usage:  0.0815 MB
👥 Duplicate Rows: 0 (0.0%)
--------------------------------------------------

🔍 Missing Value Profile:
   • Cabin: 687 missing (77.1%)
   • Age: 177 missing (19.87%)

🔗 Strong Linear Correlations (|r| >= 0.6):
   • Fare ↔ Survived (pos: 0.697)
==================================================

---

Module 2: Smart Column Suggestions

dp.suggest(df)

Analyses every column and returns actionable, rule-based preprocessing recommendations — no AI or internet required.

suggestions = dp.suggest(df)

Detects:

• 🔴 ID/key columns (100% unique values — no signal, should be dropped)
• 🔴 Constant columns (zero variance — useless for ML)
• 🔴 Columns with >60% missing data
• 🟡 Columns with 20–60% missing data
• 🟡 Unencoded categorical strings (should be One-Hot or Target encoded)
• 🟡 High-cardinality strings (need embedding or target encoding)
• 🟡 Date strings that should be parsed to datetime
• 🟡 Imbalanced binary targets (class ratio < 20%)

Console Output:

========================================================
       DATAPILOT SMART COLUMN SUGGESTIONS        
========================================================

[1] 🔴 HIGH  —  PassengerId
     Issue:      100% unique values — likely an ID or key column
     Fix:        Drop 'PassengerId' — unique identifiers leak no signal.

[2] 🟡 MEDIUM  —  Cabin
     Issue:      Very high missing rate (77.1%)
     Fix:        Consider dropping 'Cabin' — over 60% nulls will degrade model quality.

[3] 🟡 MEDIUM  —  Sex
     Issue:      Unencoded categorical string (2 unique values)
     Fix:        Encode 'Sex' using One-Hot Encoding before modelling.
========================================================

Returns: List[Dict] with keys column, issue, severity, suggestion.

---

Module 3: Dataset Analysis API

All functions accept both pandas.DataFrame and polars.DataFrame.

dp.summary(df)

meta = dp.summary(df)
# Returns: {'engine_detected': 'pandas', 'rows': 891, 'columns': 12,
#           'datatypes': {...}, 'memory_usage_mb': 0.08, 'total_missing_values': 866}

dp.missing(df)

report = dp.missing(df)

column	missing_count	missing_percentage
Cabin	687	77.10%
Age	177	19.87%

dp.duplicates(df)

dups = dp.duplicates(df)
# Returns: {'duplicate_count': 5, 'duplicate_percentage': 0.56}

dp.correlation(df, threshold=0.6)

corr = dp.correlation(df, threshold=0.6)
print(corr['strong_positive'])  # [('Fare ↔ Survived', 0.697)]
print(corr['strong_negative'])  # [('Pclass ↔ Fare', -0.549)]

---

Module 4: Outlier Detection

dp.outliers(df, method="both", z_threshold=3.0, iqr_multiplier=1.5)

Detects outliers across all numeric columns using IQR fencing and/or Z-score — whichever flags more values when method="both".

result = dp.outliers(df)
result = dp.outliers(df, method="iqr")          # IQR only
result = dp.outliers(df, method="zscore")        # Z-score only
result = dp.outliers(df, iqr_multiplier=3.0)     # extreme outliers only

Console Output:

========================================================
        DATAPILOT OUTLIER DETECTION REPORT        
  Method: BOTH | Z-threshold: 3.0 | IQR×1.5
========================================================

  🔴 HIGH  —  Fare
     Outlier count:  12 (6.7% of non-null values)
     IQR fences:     [-26.18, 65.63]
     Extreme values: [512.3, 263.0, 211.3]

Returns: Dict[column → {count, percentage, severity, values, lower_fence, upper_fence}]

---

Module 5: Auto Data Cleaning

dp.auto_clean(df, drop_null_threshold=0.6, impute_strategy="auto", drop_id_columns=True, drop_constant_columns=True)

The killer feature. Automatically fixes your dataset and tells you exactly what it changed.

clean_df, change_log = dp.auto_clean(df)

What it does (in order):

1. Drops columns where all values are the same (zero variance)
2. Drops ID/key columns where every row is unique
3. Drops columns with ≥60% null values
4. Imputes remaining numeric nulls with median
5. Imputes remaining categorical nulls with mode

Console Output:

==========================================================
         DATAPILOT AUTO-CLEAN ENGINE              
==========================================================
  Input:  891 rows × 12 columns

  🗑️  DROP   'PassengerId'  →  ID-like column (all 891 values unique)
  🗑️  DROP   'Cabin'        →  High null rate (77.1%) exceeds threshold (60%)
  🔧  IMPUTE 'Age'          →  filled 177 null(s) with median=28.0
  🔧  IMPUTE 'Embarked'     →  filled 2 null(s) with mode='S'

----------------------------------------------------------
  Output: 891 rows × 10 columns
  ✅ 2 column(s) dropped  |  2 column(s) imputed
==========================================================

Returns: Tuple[cleaned_DataFrame, List[change_log_dicts]]

---

Module 6: Train/Test Drift Detection

dp.compare(df_train, df_test, threshold=0.1)

Detects distribution shift between training and production data — the root cause of silent model degradation.

flags = dp.compare(df_train, df_test)
flags = dp.compare(df_train, df_test, threshold=0.15)  # stricter

How it works:

• Numeric columns: Flags if mean shifts >15% or std shifts >25%
• Categorical columns: Computes Jensen-Shannon divergence (0 = identical, 1 = completely different)

Console Output:

============================================================
       DATAPILOT TRAIN vs TEST DRIFT REPORT       
  Train shape: 712 rows × 11 cols
  Test  shape: 179 rows × 11 cols
  Shared columns analysed: 11
------------------------------------------------------------

  ⚠️  Age  [🟡 MEDIUM]  (numeric drift)
      Mean:  train=29.64  →  test=30.27  (2.1% shift)
      Std:   train=14.52  →  test=15.11

  ⚠️  Embarked  [🔴 HIGH]  (categorical drift)
      JS Divergence: 0.341 (threshold: 0.1)
      Train top value: 'S'  |  Test top value: 'C'

Returns: List[Dict] of flagged columns with drift statistics.

---

Module 7: Visualization Engine

One-liner plots, no matplotlib syntax required.

dp.hist(df, "Age", bins=15)      # distribution with KDE overlay
dp.box(df, "Fare")               # quartiles and outliers
dp.heatmap(df)                   # annotated Pearson correlation grid

---

Module 8: Machine Learning Diagnostics

dp.classification_report(y_true, y_pred, average="auto")

Auto-detects binary vs multi-class and picks the right averaging strategy.

metrics = dp.classification_report(y_test, predictions)
# Returns: {'accuracy': 0.85, 'precision': 0.82, 'recall': 0.88, 'f1_score': 0.85}

dp.regression_report(y_true, y_pred)

metrics = dp.regression_report(y_test, predictions)
# Returns: {'mae': 2.14, 'mse': 7.32, 'rmse': 2.71, 'r2': 0.94, 'mape': 4.8, 'max_error': 8.3}

Console Output:

==================================================
     DATAPILOT REGRESSION METRICS REPORT     
==================================================
  MAE        (Mean Absolute Error)   : 2.1400
  MSE        (Mean Squared Error)    : 7.3200
  RMSE       (Root Mean Sq. Error)   : 2.7055
  R²         (Coefficient of Det.)   : 0.9400
  MAPE       (Mean Abs. % Error)     : 4.80%
  Max Error  (Worst single pred.)    : 8.3000
--------------------------------------------------
  Verdict: ✅ EXCELLENT — model explains ≥90% of variance.
==================================================

dp.diagnose(train_score, test_score, metric_name="Accuracy")

dp.diagnose(train_score=0.98, test_score=0.72)
# → 🚨 OVERFITTING DETECTED: Add regularization, gather more data.

dp.diagnose(train_score=0.50, test_score=0.48)
# → ⚠️ UNDERFITTING DETECTED: Increase model complexity.

---

Module 9: Standalone HTML Dashboard

dp.dashboard(df, output_path="datapilot_report.html")

Generates a complete, offline-ready HTML report with:

• Summary metric cards (rows, columns, memory, duplicates, missing values)
• Data type overview table
• Missing values table + inline bar chart
• Pearson correlation pairs table
• Inline correlation heatmap image (base64 encoded — no external dependencies)

dp.dashboard(df, output_path="my_project_report.html")
# → 🎉 Standalone HTML Dashboard successfully exported to: my_project_report.html

✅ No internet required. No backend server. Just open the .html file in any browser.

---

Module 10: Performance Benchmark

dp.benchmark(df)

Proves DataPilot's Polars-native speed advantage against equivalent Pandas operations.

dp.benchmark(df)

Console Output:

==================================================================
          DATAPILOT BENCHMARK  —  Polars vs Pandas          
  Dataset: 891 rows × 12 columns
==================================================================
  Operation                   DataPilot       Pandas    Speedup
------------------------------------------------------------------
  Null Count                     0.09ms      0.48ms      5.3× ▓▓▓▓▓
  Duplicate Detection            0.11ms      0.91ms      8.3× ▓▓▓▓▓▓▓▓
  Describe / Summary Stats       0.22ms      1.47ms      6.7× ▓▓▓▓▓▓
  Correlation Matrix             0.31ms      2.18ms      7.0× ▓▓▓▓▓▓▓
  Group-By Mean                  0.08ms      0.54ms      6.8× ▓▓▓▓▓▓
------------------------------------------------------------------
  Average Speedup                                         6.8×

  ⚡ DataPilot is on average 6.8× faster than Pandas on this dataset.
  📈 Speedup grows significantly on larger datasets (1M+ rows).
==================================================================

---

Module 11: AI Copilot — 5 Providers

DataPilot uses a Metadata-Only AI Pattern — raw data rows are never transmitted to any provider. Only lightweight statistical summaries (e.g. "Age has 19.8% nulls") are sent.

Supported Providers

Provider	Type	Default Model	Requires
ollama	🔒 Local / Private	llama3	Ollama daemon running locally
openai	☁️ Cloud	gpt-4o-mini	pip install datapilot-polars[openai] + API key
gemini	☁️ Cloud	gemini-1.5-flash	pip install datapilot-polars[gemini] + API key
claude	☁️ Cloud	claude-3-haiku-20240307	pip install datapilot-polars[claude] + API key
groq	☁️ Cloud (free tier)	llama3-70b-8192	pip install datapilot-polars[groq] + API key

Usage

# 🔒 Local — fully private, no API key needed (default)
dp.analyze(df, use_ai=True)
dp.analyze(df, use_ai=True, ai_provider="ollama", ai_model="mistral")

# ☁️ OpenAI — GPT-4o, GPT-4, GPT-3.5
dp.analyze(df, use_ai=True, ai_provider="openai", api_key="sk-...")
dp.analyze(df, use_ai=True, ai_provider="openai", ai_model="gpt-4o", api_key="sk-...")

# ☁️ Google Gemini — Gemini 1.5 Pro / Flash
dp.analyze(df, use_ai=True, ai_provider="gemini", api_key="AIza...")
dp.analyze(df, use_ai=True, ai_provider="gemini", ai_model="gemini-1.5-pro", api_key="AIza...")

# ☁️ Anthropic Claude — Claude 3.5 Sonnet / Haiku
dp.analyze(df, use_ai=True, ai_provider="claude", api_key="sk-ant-...")
dp.analyze(df, use_ai=True, ai_provider="claude", ai_model="claude-3-5-sonnet-20241022", api_key="sk-ant-...")

# ☁️ Groq — Ultra-fast free-tier inference (Llama3-70b, Mixtral, Gemma2)
dp.analyze(df, use_ai=True, ai_provider="groq", api_key="gsk_...")
dp.analyze(df, use_ai=True, ai_provider="groq", ai_model="mixtral-8x7b-32768", api_key="gsk_...")

Install Only What You Need

pip install datapilot-polars[openai]    # OpenAI only
pip install datapilot-polars[gemini]    # Google Gemini only
pip install datapilot-polars[claude]    # Anthropic Claude only
pip install datapilot-polars[groq]      # Groq only (free tier)
pip install datapilot-polars[all-ai]    # All cloud providers at once

AI Output Example

🤖 AI Copilot Insights  [GROQ]:
• The Age column has 19.8% missing values — impute with median before
  training to avoid biased estimates in tree models.
• Sex and Embarked are unencoded strings — apply One-Hot Encoding;
  Sex has only 2 values making it ideal for binary encoding.
• The strong positive correlation between Fare and Survived (r=0.697)
  suggests Fare is a strong predictor — include it and check for
  outliers before normalising.

💡 Groq is recommended for getting started — it has a generous free tier, runs Llama3-70b at extremely low latency, and requires just a free account at console.groq.com.

---

Installation & Setup

Basic Setup

git clone https://github.com/yourusername/datapilot.git
cd datapilot

uv venv
source .venv/bin/activate
uv pip install -e .

With Cloud AI Providers

uv pip install -e .[openai]     # Add OpenAI support
uv pip install -e .[groq]       # Add Groq support (free)
uv pip install -e .[all-ai]     # Add all cloud AI providers

Development Setup (with tests)

uv pip install -e .[dev]
pytest

Initializing Local AI (Ollama)

Only needed if you use ai_provider="ollama" (the default):

# Start the Ollama daemon
ollama serve

# Pull a model (first time only)
ollama pull llama3

---

Troubleshooting

ModuleNotFoundError after editing files

uv pip install -e . --force-reinstall

AI: Ollama Connection Refused

ollama serve   # make sure the Ollama daemon is running

AI: Missing cloud provider package

# Install the extra for your chosen provider
pip install datapilot-polars[openai]   # or gemini / claude / groq / all-ai

AI: Invalid or missing API key

# Always pass the key explicitly for cloud providers
dp.analyze(df, use_ai=True, ai_provider="openai", api_key="sk-...")

Running Tests

pytest -v
pytest --cov=datapilot   # with coverage report

---

Contributing

See CONTRIBUTING.md for development setup, code style guidelines, and the pull request process.

---

Changelog

See CHANGELOG.md for a full version history.

---

License

MIT License — see LICENSE for details.