FlashEDA

Current version: 0.5.0 · Changelog

Analyse any dataset in a few seconds, then get validated cleaning, feature and training code that runs on the full data, in the engine that fits its size: pandas, polars, DuckDB or PySpark.

FlashEDA analyses a fixed-size random sample, so analysis time barely changes between 100 thousand and 100 million rows.

Install

pip install flasheda            # core
pip install "flasheda[fast]"    # + polars, DuckDB, pyarrow (recommended for big files)
pip install "flasheda[ml]"      # + scikit-learn, for flasheda.train
pip install "flasheda[sql]"     # + SQLAlchemy, for database tables
pip install "flasheda[spark]"   # + PySpark
pip install "flasheda[all]"     # everything except PySpark

Quick start

import flasheda as fe

# 1. Analyse: any size, a few seconds
report = fe.analyze("orders.csv", target="churn", save_pdf=False)
report.show()                      # console report
report.save_html("report.html")    # or report.save_pdf("report.pdf")

# 2. Clean: get a script for the full data, already checked on the sample
code = fe.clean("orders.csv", target="churn", engine="polars")
code.save("clean_orders.py")       # then: python clean_orders.py

# 3. Model: compare models on the sample, get prep + training scripts
result = fe.train("orders.csv", target="churn", engine="polars")
result.show()                      # leaderboard
result.save("prep.py", "train.py") # python prep.py && python train.py → model.joblib

# 4. Guard: learn rules from good data, check every new file
contract = fe.analyze("orders_jan.csv").contract()
fe.check("orders_feb.csv", contract).show()
open("check.py", "w").write(contract.script())   # python check.py new.csv → exit 1 on failure

# 5. Compare: what changed between two datasets
fe.compare("orders_jan.csv", "orders_feb.csv").show()

# Databases work like files
src = fe.sql("postgresql://user:pass@host/shop", table="orders")
fe.analyze(src, target="churn")

What it does

Cleaning steps

Each step in the generated code has a comment saying why it was added:

• drop likely-ID, mostly-empty, constant and leaky columns
• drop rows where the target is missing, and drop duplicate rows
• convert numbers stored as text ("12.5", "N/A") to numbers
• merge spelling variants ("Delhi", " delhi", "DELHI")
• group rare categories into other
• add <col>_was_missing flags when missingness predicts the target
• fill nulls (median / mode), log-transform skewed columns, clip outliers

Features and models

fe.train() (or report.train()) adds features on top of the cleaning steps, compares models on the sample, and writes two scripts:

• prep.py: cleaning + features for the full data, in the chosen engine
• train.py: scikit-learn training with cross-validation, using the same metric and split, that saves model.joblib

The metric and split follow the target: PR-AUC for imbalanced binary targets, MAE for skewed regression targets, time-based splits when there's a date column, and so on.

Data contracts and drift

A contract describes what good data looks like, learned from a sample with tolerances so normal variation passes: type, missing %, 1st/99th percentile range (catches unit changes like ×100), no negatives, allowed values, uniqueness of IDs. fe.check() checks every row of a file in one DuckDB pass; contract.script() writes the same check as a standalone script that exits with code 1 on failure, ready for a pipeline or CI job.

fe.compare(old, new) reports added/removed columns, type changes and how much each column shifted (PSI: < 0.1 stable, 0.1–0.25 moderate, ≥ 0.25 major). Dates and IDs are shown as range shifts, since they always move.

Engine choice

Override with fe.analyze(..., engine="duckdb") or report.codegen("duckdb").

Performance

Measured on synthetic messy data:

Examples

• examples/phase1_demo.py: engine choice and code generation at 0.1M / 1M / 5M rows
• examples/phase2_demo.py: target analysis and cleaning on a 1M-row churn CSV
• examples/phase3_demo.py: features, model comparison and training on 1M rows
• examples/phase4_demo.py: contracts, check script, drift and a database on 1M rows

Development

pip install -e ".[dev]"
pytest

License

MIT