frameprep
Automated ML-Ready Data Preparation Library
One pipeline. Raw DataFrame in. ML-ready DataFrame out.
The Problem
Data teams at mid-size companies spend 60–70 % of project time cleaning and
preparing raw data before any model can touch it. frameprep automates the
most repetitive and error-prone parts of that work.
What It Does
| Module | What It Handles |
|---|---|
| Smart dtype inference | Detects real types (bool stored as string, datetime in text, numeric strings) and downcasts to smallest memory-safe type |
| Missing value strategy | Runs statistical tests (MCAR vs MAR proxy) to pick the right imputer per column — mean, median, KNN, mode, or sentinel |
| Encoding + scaling | Label, OHE, or hashing for categoricals; standard / minmax / robust for numerics |
| Leakage detection + audit | Flags correlated, near-constant, and ID-like columns; produces a full JSON/DataFrame audit report |
Installation
pip install frameprep
For development:
git clone https://github.com/rahulreddy9725/frameprep.git
cd frameprep
pip install -e ".[dev]"
Quick Start
import pandas as pd
from frameprep import frameprepPipeline
df_raw = pd.read_csv("customers.csv")
pipeline = frameprepPipeline(
target_col="churn", # excluded from encoding/scaling
scaling_strategy="standard", # 'standard' | 'minmax' | 'robust'
correlation_threshold=0.95, # leakage flag threshold
verbose=True,
)
df_clean, report = pipeline.fit_transform(df_raw)
# Inspect the audit
report.summary() # prints to stdout
report.to_json() # full JSON string
report.to_dataframe() # flat pandas DataFrame
report.save("audit.json") # write to disk
# Apply the same transforms to test/prod data
df_test_clean = pipeline.transform(df_test)
Audit Report Example
============================================================
frameprep — Preparation Audit Report
============================================================
Created at : 2024-01-15T10:23:45+00:00
Elapsed : 0.83s
Shape
Input : 5000 rows × 18 columns
Output : 5000 rows × 24 columns
Δ cols : +6
Memory Optimisation
Before : 720.0 KB
After : 218.4 KB
Saved : 69.7%
Imputation Strategies
median : 4 column(s)
knn : 2 column(s)
mode : 3 column(s)
none : 9 column(s)
Encoding Strategies
one_hot_encode : 3 column(s)
label_encode : 2 column(s)
feature_hash : 1 column(s)
Scaling Strategy : standard (8 columns)
Leakage Detected : YES ⚠
• row_id: monotonically increasing integer — likely a row ID
============================================================
Configuration Reference
frameprepPipeline(
target_col=None, # str — label column, excluded from transforms
exclude_cols=[], # list — columns to pass through unchanged
correlation_threshold=0.95, # float — leakage correlation cutoff
cardinality_threshold=50, # int — above this → hashing, below → OHE
scaling_strategy="standard", # str — 'standard' | 'minmax' | 'robust'
verbose=True, # bool — step-by-step logging
)
Module Details
1. DTypeInferrer
- Detects boolean strings (
"yes"/"no","true"/"false") - Detects datetime strings (
"2023-01-15","15/01/2023") - Detects numeric strings (
"100","3.14") - Downcasts
float64 → float32,int64 → int8/int16/int32 - Low-cardinality objects →
pd.Categorical
2. MissingValueHandler
| Condition | Strategy |
|---|---|
| 0 % missing | skip |
| < 1 % | median / mode (fast) |
| MAR detected (missingness correlated with other columns) | KNN imputer |
| |skew| > 1 | median |
| |skew| ≤ 1 | mean |
| > 40 % missing | constant flag (-999) |
| Categorical < 30 % | mode |
| Categorical ≥ 30 % | "__missing__" sentinel |
3. CategoricalEncoder
| Cardinality | Strategy |
|---|---|
| 2 unique values | LabelEncoder |
| 3 – threshold | OneHotEncoder (drop first) |
| > threshold | FeatureHasher |
| Boolean dtype | cast to 0/1 |
4. LeakageDetector
Runs three checks:
- Near-constant: > 99 % of values identical
- High correlation with target: Pearson |r| ≥ threshold
- ID-like column: monotonically increasing integer
Development
# Run tests
pytest
# Run tests with coverage
pytest --cov=frameprep --cov-report=term-missing
# Lint
ruff check frameprep tests
# Type check
mypy frameprep
Project Structure
frameprep/
├── frameprep/
│ ├── __init__.py
│ ├── pipeline.py # Main frameprepPipeline
│ ├── core/
│ │ ├── dtype_inferrer.py
│ │ └── missing_strategy.py
│ ├── encoders/
│ │ └── categorical.py
│ ├── scalers/
│ │ └── numeric.py
│ ├── detectors/
│ │ └── leakage.py
│ ├── reports/
│ │ └── audit.py
│ └── utils/
│ ├── logger.py
│ └── validators.py
├── tests/
│ ├── conftest.py
│ ├── unit/
│ │ ├── test_dtype_inferrer.py
│ │ ├── test_missing_strategy.py
│ │ ├── test_categorical_encoder.py
│ │ ├── test_numeric_scaler.py
│ │ └── test_leakage_detector.py
│ └── integration/
│ └── test_pipeline.py
├── docs/examples/
│ └── quickstart.ipynb
├── pyproject.toml
├── README.md
├── CONTRIBUTING.md
├── CHANGELOG.md
└── LICENSE
License
MIT — see LICENSE.
Metadata
Release files for frameprep 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| frameprep-0.1.0.tar.gz | 22.5 kB | Without provenance | Details for frameprep-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| frameprep-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for frameprep-0.1.0-py3-none-any.whl |
Total release size: 47.2 kB
Release files / frameprep-0.1.0.tar.gz
| Download URL | frameprep-0.1.0.tar.gz |
|---|---|
| Size | 22.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
68b7f860645953b85a151991ef681c6e3e08e0c4a90cd8e94a1c8b191ee387c4
|
|
BLAKE2b-256 checksum How to use checksums |
99e7a516e1e6a9ecca80c2ce57faade9425db9a017ab147fcdc914a2de968221
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.11
|
Release files / frameprep-0.1.0-py3-none-any.whl
| Download URL | frameprep-0.1.0-py3-none-any.whl |
|---|---|
| Size | 24.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b79150a78abdea7325ef372cab5c9e9e0ffa514a54efd8d85b0c035aa33ac823
|
|
BLAKE2b-256 checksum How to use checksums |
f087c0350f0104ac63276135599059d9462f179164dd58842b0fbf6943e1d4b9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.11
|
Security for frameprep 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.