frameprep 0.1.0

Automated ML-Ready Data Preparation Library

frameprep

Automated ML-Ready Data Preparation Library

One pipeline. Raw DataFrame in. ML-ready DataFrame out.

---

The Problem

Data teams at mid-size companies spend 60–70 % of project time cleaning and preparing raw data before any model can touch it. frameprep automates the most repetitive and error-prone parts of that work.

---

What It Does

Module	What It Handles
Smart dtype inference	Detects real types (bool stored as string, datetime in text, numeric strings) and downcasts to smallest memory-safe type
Missing value strategy	Runs statistical tests (MCAR vs MAR proxy) to pick the right imputer per column — mean, median, KNN, mode, or sentinel
Encoding + scaling	Label, OHE, or hashing for categoricals; standard / minmax / robust for numerics
Leakage detection + audit	Flags correlated, near-constant, and ID-like columns; produces a full JSON/DataFrame audit report

---

Installation

pip install frameprep

For development:

git clone  https://github.com/rahulreddy9725/frameprep.git
cd frameprep
pip install -e ".[dev]"

---

Quick Start

import pandas as pd
from frameprep import frameprepPipeline

df_raw = pd.read_csv("customers.csv")

pipeline = frameprepPipeline(
    target_col="churn",          # excluded from encoding/scaling
    scaling_strategy="standard", # 'standard' | 'minmax' | 'robust'
    correlation_threshold=0.95,  # leakage flag threshold
    verbose=True,
)

df_clean, report = pipeline.fit_transform(df_raw)

# Inspect the audit
report.summary()                 # prints to stdout
report.to_json()                 # full JSON string
report.to_dataframe()            # flat pandas DataFrame
report.save("audit.json")        # write to disk

# Apply the same transforms to test/prod data
df_test_clean = pipeline.transform(df_test)

---

Audit Report Example

============================================================
  frameprep — Preparation Audit Report
============================================================
  Created at   : 2024-01-15T10:23:45+00:00
  Elapsed      : 0.83s

  Shape
    Input  : 5000 rows × 18 columns
    Output : 5000 rows × 24 columns
    Δ cols : +6

  Memory Optimisation
    Before : 720.0 KB
    After  : 218.4 KB
    Saved  : 69.7%

  Imputation Strategies
    median              : 4 column(s)
    knn                 : 2 column(s)
    mode                : 3 column(s)
    none                : 9 column(s)

  Encoding Strategies
    one_hot_encode      : 3 column(s)
    label_encode        : 2 column(s)
    feature_hash        : 1 column(s)

  Scaling Strategy   : standard (8 columns)

  Leakage Detected   : YES ⚠
    • row_id: monotonically increasing integer — likely a row ID
============================================================

---

Configuration Reference

frameprepPipeline(
    target_col=None,              # str  — label column, excluded from transforms
    exclude_cols=[],              # list — columns to pass through unchanged
    correlation_threshold=0.95,  # float — leakage correlation cutoff
    cardinality_threshold=50,    # int  — above this → hashing, below → OHE
    scaling_strategy="standard", # str  — 'standard' | 'minmax' | 'robust'
    verbose=True,                # bool — step-by-step logging
)

---

Module Details

1. DTypeInferrer

• Detects boolean strings ("yes"/"no", "true"/"false")
• Detects datetime strings ("2023-01-15", "15/01/2023")
• Detects numeric strings ("100", "3.14")
• Downcasts float64 → float32, int64 → int8/int16/int32
• Low-cardinality objects → pd.Categorical

2. MissingValueHandler

Condition	Strategy
0 % missing	skip
< 1 %	median / mode (fast)
MAR detected (missingness correlated with other columns)	KNN imputer
|skew| > 1	median
|skew| ≤ 1	mean
> 40 % missing	constant flag (-999)
Categorical < 30 %	mode
Categorical ≥ 30 %	"__missing__" sentinel

3. CategoricalEncoder

Cardinality	Strategy
2 unique values	LabelEncoder
3 – threshold	OneHotEncoder (drop first)
> threshold	FeatureHasher
Boolean dtype	cast to 0/1

4. LeakageDetector

Runs three checks:

• Near-constant: > 99 % of values identical
• High correlation with target: Pearson |r| ≥ threshold
• ID-like column: monotonically increasing integer

---

Development

# Run tests
pytest

# Run tests with coverage
pytest --cov=frameprep --cov-report=term-missing

# Lint
ruff check frameprep tests

# Type check
mypy frameprep

---

Project Structure

frameprep/
├── frameprep/
│   ├── __init__.py
│   ├── pipeline.py          # Main frameprepPipeline
│   ├── core/
│   │   ├── dtype_inferrer.py
│   │   └── missing_strategy.py
│   ├── encoders/
│   │   └── categorical.py
│   ├── scalers/
│   │   └── numeric.py
│   ├── detectors/
│   │   └── leakage.py
│   ├── reports/
│   │   └── audit.py
│   └── utils/
│       ├── logger.py
│       └── validators.py
├── tests/
│   ├── conftest.py
│   ├── unit/
│   │   ├── test_dtype_inferrer.py
│   │   ├── test_missing_strategy.py
│   │   ├── test_categorical_encoder.py
│   │   ├── test_numeric_scaler.py
│   │   └── test_leakage_detector.py
│   └── integration/
│       └── test_pipeline.py
├── docs/examples/
│   └── quickstart.ipynb
├── pyproject.toml
├── README.md
├── CONTRIBUTING.md
├── CHANGELOG.md
└── LICENSE

---

License

MIT — see LICENSE.