edaprep

Transparent, leakage-safe EDA and ML preprocessing, with an explainable planner.

edaprep looks at a dataset, works out which preprocessing operations actually apply to it, tells you what it intends to do and why, and then does it — fitting every statistic on the training data alone.

import edaprep

pipe = edaprep.AutoPipeline(target="churn", model_family="tree", random_state=42)
pipe.fit(train_df)
pipe.explain()

X_train = pipe.transform(train_df)
X_test  = pipe.transform(test_df)

income:
  + outliers_report - skew 3.22 is moderate (>= 1.0); IQR fence widened to k=3.0 for the asymmetry
  + impute_median - 2.0% missing; median rather than mean because it is unaffected by
                    the skew (3.22) and by outliers
  + transform_log1p - skew 3.22 is moderate and the column is non-negative (min 2576.27),
                      so log1p applies and is invertible
  + scale_robust - skew 3.22; robust scaling (median and IQR) rather than standard,
                   whose standard deviation is dominated by the tail

city:
  + group_rare_categories - 163 levels; those appearing in fewer than 5 rows (1.0%) are
                            grouped, since they cannot support a reliable estimate
  + encode_target - 163 levels exceeds the 50-level one-hot ceiling; target encoding is
                    used with 5-fold cross-fitting so no row is encoded using its own target

customer_id:
  x dropped - identifier: 100.0% of values are distinct, so it cannot generalise beyond
              the rows it was fitted on

Why it exists

Tabular ML notebooks converge on nearly the same workflow regardless of domain — and they go wrong in the same places. docs/design-rationale.md catalogues that workflow and names the failure modes; each one shaped a design decision here:

• The dtype-based column split (select_dtypes(include=['int64','float64'])) is the most-typed line in tabular data science and the largest single source of error: it sends a zip code and a temperature down the same path, and drops a numeric column stored as text entirely. edaprep infers a semantic type and reports its confidence.
• The IQR and z-score fences get rewritten in project after project, with the multiplier drifting between 1.5 and 3.0 for no recorded reason. Both are now single parameterised, named, reported operations — and the index-alignment bug that silently flags the wrong rows when a column has any NaN is fixed once, here.
• Leakage is easy to introduce and hard to notice. Fitting a StandardScaler on the full frame, writing the result to CSV, then splitting afterwards looks fine and poisons every downstream experiment. edaprep makes it structurally impossible rather than merely discouraged.
• Sophisticated pipelines end up routing, not sequencing — branching on skewness for numerics and on the consuming model for categoricals, usually buried in a make_preprocessor closure. That became the planner, and model_family.

Installation

pip install edaprep                    # core: numpy, pandas, scipy
pip install "edaprep[visualization]"   # + matplotlib
pip install "edaprep[advanced]"        # + scikit-learn
pip install "edaprep[all]"

Python 3.9–3.13, tested on Linux, macOS and Windows.

Would rather try it than install it? There is a runnable notebook on Kaggle that needs no setup — it measures two preprocessing leaks on Telco Churn, one worth +0.000001 and one scoring a perfect 1.00000 AUC on pure noise, then shows what this library does about them.

What it does

Understand a dataset

profile = edaprep.profile(df, target="churn")
print(profile.summary())

Dataset
  600 rows x 18 columns
  300.3 KB in memory
  628 missing cells (5.81%)
  target: churn (classification, 2 classes, minority/majority ratio 0.232)

Semantic types
  numeric           5
  binary            4
  categorical       3
  ...

Data-quality findings
  [x] 1 column(s) are almost perfectly associated with the target (>= 0.98). This
      usually means the column encodes the answer: 'leaky'.
  [!] 2 column(s) contain placeholder strings that most likely mean 'missing' but are
      not recognised as NaN: 'workclass', 'occupation'.
  [!] 1 group(s) of identical columns: income=income_copy
  [i] 1 column pair(s) go missing together, which usually means a shared cause:
      income~income_copy (1.00)

Explore it

report = edaprep.EDA(df, target="churn").analyze("standard")
print(report.summary())
report.numerical        # a DataFrame
report.to_html("eda.html")

Three levels that differ in work done, not just in what is shown: quick skips every O(n log n) and O(p²) computation, standard adds moments, outliers, correlation and target relationships, deep adds VIF and significance tests with a Benjamini-Hochberg adjustment.

Prepare it

pipe = edaprep.AutoPipeline(target="churn", model_family="linear", random_state=42)
X_train = pipe.fit_transform(train_df)
X_test  = pipe.transform(test_df)

pipe.plan_             # the decisions, serialisable and editable
pipe.report_           # what actually happened, with counts
pipe.transformations_  # one row per decision, as a DataFrame
pipe.statistics_       # every learned parameter

Or say exactly what should happen:

pipe = (
    edaprep.Pipeline(target="churn")
    .flag_missing()
    .handle_outliers(strategy="clip")
    .handle_missing()
    .encode_categorical()
    .scale_numeric()
)

Override anything

config = edaprep.Config(random_state=42)
config.column("age").imputation = "mean"
config.column("income").outlier_strategy = "clip"
config.column("city").encoding = "frequency"
config.column("zip").semantic_type = "categorical"
config.thresholds.skew_heavy = 4.0

pipe = edaprep.AutoPipeline(target="churn", config=config)

Overrides are tagged in the plan, so explain() marks them as yours rather than presenting them as the planner's reasoning.

Design guarantees

No leakage, structurally. Learned state lives only in attributes written inside fit; transform is a pure function of that state. The property is asserted directly: a test transforms a frame whole and then row by row and requires identical output, which fails immediately if anything recomputes a statistic at transform time.

Nothing silent. Dropped columns, imputed values, grouped categories, clipped rows and unseen categories are all counted and reported. edaprep never calls warnings.filterwarnings.

Everything explainable. Every automatic decision carries an English rationale naming the measurement behind it. The plan is inert, serialisable data — printable, diffable, storable next to a model artefact, and re-executable.

Reproducible. random_state seeds every stochastic step. The report records the library version, the configuration, the seed, whether profiling sampled, and every learned parameter.

Conservative. Outliers are reported, not deleted, by default. Duplicate rows are reported, not removed — repeated observations are legitimate in transactional data. Class imbalance is measured and reported; resampling is a modelling decision that belongs after the split, so edaprep does not do it.

Performance

Measured, not asserted. See docs/performance.md.

100,000 rows unless stated. The most instructive result is one that went the other way: a hand-written NumPy kernel in this library turned out to be 2.1× slower than the pandas code it replaced, so it was deleted. That story is in docs/performance.md §1.

No native code. Nothing here is un-vectorisable, and the one place a hand-written kernel looked promising was slower than pandas.

Documentation

Scope

In: dataset inspection, EDA, data quality, cleaning, missing values, duplicates, outliers, dtype inference, categorical encoding, numeric transformation, scaling, feature selection, datetime expansion, leakage-safe train/test preparation, pipelines, reporting.

Out, deliberately: model training, resampling, hyperparameter search, NLP, forecasting, deep learning, distributed execution. Extension points exist for each (docs/extending.md), and none is implemented in v1.

Contributing

Contributions are welcome, and CONTRIBUTING.md is written to make the first one straightforward: it explains the architecture in a page, lists the rules CI actually enforces, and points at issues scoped so that each one names the file to change and the test to write.

Issues labelled good first issue are a deliberate starting set. Comment on one to claim it before you start — two people once fixed the same issue eight hours apart, and one of them had to be turned away.

pip install -e ".[dev]"
pytest                       # 367 tests, ~15s
ruff check src/ tests/ benchmarks/ examples/
python benchmarks/bench.py

Contributors

This library argues that a decision is worth little without the reasoning behind it, so it would be odd to credit only the diffs. Measurement, review and design that changed what shipped are listed here alongside the commits.

GitHub's contributors graph counts commits on main, so it cannot show a benchmark that settled an argument or a review that caught a bug. This table can. If you contributed something that changed the library and you are not on it, that is an oversight worth an issue — please open one.

Licence

MIT.