Skip to main content

PreToolkit

CI PyPI

Composable, reproducible, and maintainable data preprocessing for machine learning.

PreToolkit provides data preparation steps — imputation, encoding, scaling, and more — as self-contained building blocks that you chain into a pipeline. Instead of rewriting the same cleaning code in every project, you compose blocks that keep track of how they transform your data and guarantee the same input is always transformed the same way.

Part of the Octacity community · built by the .LUCID team.

Status: v0.1.0, the first release. The core building blocks (MissingValueImputer, CategoricalEncoder, FeatureScaler) and Pipeline are in place; the API may still change before 1.0. See the quickstart notebook for a walkthrough.


Why

In almost every ML project, a large share of the work is preparing the data. Yet that code is usually rewritten from scratch, slightly differently each time, and it's easy to fall into traps — most commonly data leakage, where information from the test set silently influences training.

scikit-learn gives you the primitives, but leaves reproducibility, data-type awareness, and feature-name tracking up to you. PreToolkit is a higher-level layer that makes those three things central:

  • Reproducibility first — every block records its fitted state and the exact transformation applied, so the same dataset yields identical output across runs and people.
  • Data-type awareness — blocks know which columns are numeric vs. categoric and route transformations automatically; no manual column slicing.
  • Metadata tracking — the toolkit tracks how input feature names map to output feature names across the whole chain, even after one-hot encoding or binning.

This is not a replacement for scikit-learn — it is designed to work alongside it: blocks run inside scikit-learn pipelines, cross_val_score and GridSearchCV, without depending on it.


Installation

pip install pretoolkit

For development, clone the repository and let uv create the environment (adds pytest, ruff, scikit-learn and notebook tooling):

git clone https://github.com/octacity-org/PreToolkit.git
cd PreToolkit
uv sync

Requires Python 3.10+. Runtime dependencies: numpy, pandas. scikit-learn is not required at runtime, but the blocks work inside scikit-learn pipelines, clone, cross_val_score and GridSearchCV.


Quick start

The example below creates a small inline dataset with numeric, categorical, and missing values, and processes it through a complete pipeline:

import numpy as np
import pandas as pd
from pretoolkit import (
    CategoricalEncoder,
    FeatureScaler,
    MissingValueImputer,
    Pipeline,
)

# Training and test data with numeric, categorical, and missing values
X_train = pd.DataFrame(
    {
        "age": [25.0, np.nan, 35.0, 40.0],
        "city": ["Paris", "London", "Paris", np.nan],
        "salary": [50000.0, 60000.0, np.nan, 80000.0],
    }
)

X_test = pd.DataFrame(
    {
        "age": [30.0, 45.0],
        "city": ["London", "Paris"],
        "salary": [55000.0, 75000.0],
    }
)

pipe = Pipeline(
    [
        MissingValueImputer(strategy="median"),
        CategoricalEncoder(method="onehot"),
        FeatureScaler(method="standard"),
    ]
)

# Fit on training data only, then transform both — no data leakage
X_train_clean = pipe.fit_transform(X_train)
X_test_clean = pipe.transform(X_test)

print(pipe.get_feature_names_out())
# ['age', 'city_London', 'city_Paris', 'salary']

print(X_train_clean.round(2))
#     age  city_London  city_Paris  salary
# 0 -1.61        -0.58        0.58   -1.15
# 1  0.23         1.73       -1.73   -0.23
# 2  0.23        -0.58        0.58   -0.23
# 3  1.15        -0.58        0.58    1.61

The key detail: fit runs on the training data only. When you call transform on the test set, the statistics learned from training are applied — the test set never leaks into the fitting process.

For an end-to-end interactive tutorial including scikit-learn cross-validation and a data leakage demonstration, see examples/quickstart.ipynb.


How it works

A single abstract base class defines the contract; every block inherits from it and shares the same interface.

BasePreprocessor (abstract)
  |-- fit()  .  transform()  .  fit_transform()   -> shared logic
  \-- _fit() .  _transform()                       -> each block implements

Concrete blocks:
  |-- MissingValueImputer   (mean / median / most_frequent)
  |-- CategoricalEncoder    (one-hot / ordinal)
  \-- FeatureScaler         (standard / minmax)

Pipeline (also a BasePreprocessor):
  \-- chains blocks, applied in sequence

Because every block shares fit / transform, the pipeline treats them uniformly, and new blocks can be added without touching existing code. Each block only implements its own _fit and _transform; the shared logic (fitted-state checks, leakage guard) lives in the base class.


Blocks

MissingValueImputer learns fill values from training data and reuses them for new data. Numeric columns use mean (the default), median or most_frequent; non-numeric columns always use their most frequent value.

from pretoolkit import MissingValueImputer

imputer = MissingValueImputer(strategy="median")
X_train_filled = imputer.fit_transform(X_train)
X_test_filled = imputer.transform(X_test)
print(imputer.statistics_)
Block What it does Options
MissingValueImputer Fills missing values mean, median, most_frequent
CategoricalEncoder Converts categories to numbers onehot, ordinal; handle_unknown="error" or "ignore"
FeatureScaler Normalizes value ranges standard, minmax
Pipeline Chains blocks in sequence —

Roadmap

v0.1 (released) — the three blocks above, a pipeline to chain them, leakage protection, unit tests, and documentation. See the changelog.

Beyond v0.1, each feature is an open contribution opportunity:

  • v0.2 (next) — ordinal encoding with custom order, robust scaler, outlier handling, applying a block to selected columns only
  • v0.3 — feature binning, interaction terms, full-chain feature-name mapping
  • v0.4+ — target / frequency / binary encoding for high-cardinality columns, pipeline save & load

Contributing

Contributions are welcome — the modular design means each block is largely independent, so you can own one without blocking anyone else.

To add a new block:

  1. Claim an existing issue by commenting on it, or open one describing the block and its interface.
  2. Subclass BasePreprocessor and implement _fit and _transform.
  3. Add unit tests and a short usage example.
  4. Open a PR; the maintaining team reviews for interface consistency.

See CONTRIBUTING.md for details.


License

MIT — see LICENSE.

Metadata

Release files for pretoolkit 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for pretoolkit 0.1.0
File Interpreter ABI Platform Actions
pretoolkit-0.1.0-py3-none-any.whl Python 3 none any Details for pretoolkit-0.1.0-py3-none-any.whl

Release files / pretoolkit-0.1.0-py3-none-any.whl

Download URL pretoolkit-0.1.0-py3-none-any.whl
Size 14.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
e17b639016af6f6d72a85094b39f25df5cc3d7587efe2d978c825ba868b93c2d
BLAKE2b-256 checksum
How to use checksums
4c27540de8f498e795cd57e9e62c570c0d371a87d16c64284d3b525f8320eef4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for pretoolkit 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

1 release file No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page