Skip to main content

frameprep

Automated ML-Ready Data Preparation Library

One pipeline. Raw DataFrame in. ML-ready DataFrame out.


The Problem

Data teams at mid-size companies spend 60–70 % of project time cleaning and preparing raw data before any model can touch it. frameprep automates the most repetitive and error-prone parts of that work.


What It Does

Module What It Handles
Smart dtype inference Detects real types (bool stored as string, datetime in text, numeric strings) and downcasts to smallest memory-safe type
Missing value strategy Runs statistical tests (MCAR vs MAR proxy) to pick the right imputer per column — mean, median, KNN, mode, or sentinel
Encoding + scaling Label, OHE, or hashing for categoricals; standard / minmax / robust for numerics
Leakage detection + audit Flags correlated, near-constant, and ID-like columns; produces a full JSON/DataFrame audit report

Installation

pip install frameprep

For development:

git clone  https://github.com/rahulreddy9725/frameprep.git
cd frameprep
pip install -e ".[dev]"

Quick Start

import pandas as pd
from frameprep import frameprepPipeline

df_raw = pd.read_csv("customers.csv")

pipeline = frameprepPipeline(
    target_col="churn",          # excluded from encoding/scaling
    scaling_strategy="standard", # 'standard' | 'minmax' | 'robust'
    correlation_threshold=0.95,  # leakage flag threshold
    verbose=True,
)

df_clean, report = pipeline.fit_transform(df_raw)

# Inspect the audit
report.summary()                 # prints to stdout
report.to_json()                 # full JSON string
report.to_dataframe()            # flat pandas DataFrame
report.save("audit.json")        # write to disk

# Apply the same transforms to test/prod data
df_test_clean = pipeline.transform(df_test)

Audit Report Example

============================================================
  frameprep — Preparation Audit Report
============================================================
  Created at   : 2024-01-15T10:23:45+00:00
  Elapsed      : 0.83s

  Shape
    Input  : 5000 rows × 18 columns
    Output : 5000 rows × 24 columns
    Δ cols : +6

  Memory Optimisation
    Before : 720.0 KB
    After  : 218.4 KB
    Saved  : 69.7%

  Imputation Strategies
    median              : 4 column(s)
    knn                 : 2 column(s)
    mode                : 3 column(s)
    none                : 9 column(s)

  Encoding Strategies
    one_hot_encode      : 3 column(s)
    label_encode        : 2 column(s)
    feature_hash        : 1 column(s)

  Scaling Strategy   : standard (8 columns)

  Leakage Detected   : YES ⚠
    • row_id: monotonically increasing integer — likely a row ID
============================================================

Configuration Reference

frameprepPipeline(
    target_col=None,              # str  — label column, excluded from transforms
    exclude_cols=[],              # list — columns to pass through unchanged
    correlation_threshold=0.95,  # float — leakage correlation cutoff
    cardinality_threshold=50,    # int  — above this → hashing, below → OHE
    scaling_strategy="standard", # str  — 'standard' | 'minmax' | 'robust'
    verbose=True,                # bool — step-by-step logging
)

Module Details

1. DTypeInferrer

  • Detects boolean strings ("yes"/"no", "true"/"false")
  • Detects datetime strings ("2023-01-15", "15/01/2023")
  • Detects numeric strings ("100", "3.14")
  • Downcasts float64 → float32, int64 → int8/int16/int32
  • Low-cardinality objects → pd.Categorical

2. MissingValueHandler

Condition Strategy
0 % missing skip
< 1 % median / mode (fast)
MAR detected (missingness correlated with other columns) KNN imputer
|skew| > 1 median
|skew| ≤ 1 mean
> 40 % missing constant flag (-999)
Categorical < 30 % mode
Categorical ≥ 30 % "__missing__" sentinel

3. CategoricalEncoder

Cardinality Strategy
2 unique values LabelEncoder
3 – threshold OneHotEncoder (drop first)
> threshold FeatureHasher
Boolean dtype cast to 0/1

4. LeakageDetector

Runs three checks:

  • Near-constant: > 99 % of values identical
  • High correlation with target: Pearson |r| ≥ threshold
  • ID-like column: monotonically increasing integer

Development

# Run tests
pytest

# Run tests with coverage
pytest --cov=frameprep --cov-report=term-missing

# Lint
ruff check frameprep tests

# Type check
mypy frameprep

Project Structure

frameprep/
├── frameprep/
│   ├── __init__.py
│   ├── pipeline.py          # Main frameprepPipeline
│   ├── core/
│   │   ├── dtype_inferrer.py
│   │   └── missing_strategy.py
│   ├── encoders/
│   │   └── categorical.py
│   ├── scalers/
│   │   └── numeric.py
│   ├── detectors/
│   │   └── leakage.py
│   ├── reports/
│   │   └── audit.py
│   └── utils/
│       ├── logger.py
│       └── validators.py
├── tests/
│   ├── conftest.py
│   ├── unit/
│   │   ├── test_dtype_inferrer.py
│   │   ├── test_missing_strategy.py
│   │   ├── test_categorical_encoder.py
│   │   ├── test_numeric_scaler.py
│   │   └── test_leakage_detector.py
│   └── integration/
│       └── test_pipeline.py
├── docs/examples/
│   └── quickstart.ipynb
├── pyproject.toml
├── README.md
├── CONTRIBUTING.md
├── CHANGELOG.md
└── LICENSE

License

MIT — see LICENSE.

Metadata

Release files for frameprep 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for frameprep 0.1.0
File Size Uploaded Provenance Actions
frameprep-0.1.0.tar.gz 22.5 kB Without provenance Details for frameprep-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for frameprep 0.1.0
File Interpreter ABI Platform Actions
frameprep-0.1.0-py3-none-any.whl Python 3 none any Details for frameprep-0.1.0-py3-none-any.whl

Total release size: 47.2 kB

Release files / frameprep-0.1.0.tar.gz

Download URL frameprep-0.1.0.tar.gz
Size 22.5 kB
Tags Source
SHA-256 checksum
How to use checksums
68b7f860645953b85a151991ef681c6e3e08e0c4a90cd8e94a1c8b191ee387c4
BLAKE2b-256 checksum
How to use checksums
99e7a516e1e6a9ecca80c2ce57faade9425db9a017ab147fcdc914a2de968221
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.11

Release files / frameprep-0.1.0-py3-none-any.whl

Download URL frameprep-0.1.0-py3-none-any.whl
Size 24.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b79150a78abdea7325ef372cab5c9e9e0ffa514a54efd8d85b0c035aa33ac823
BLAKE2b-256 checksum
How to use checksums
f087c0350f0104ac63276135599059d9462f179164dd58842b0fbf6943e1d4b9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.11

Security for frameprep 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page