fast-feature-transform

Feature preprocessing for machine learning, written in C++ with a Python API. Fit a pipeline on Parquet files, save it, then transform one record at a time when serving or whole files when training.

• Feature parity with category_encoders and scikit-learn's preprocessors.
• Fast: one record takes 0.13 to 1.2 µs, where scikit-learn takes 32 to 219 µs and category_encoders 0.31 to 1.2 ms.
• Out-of-core execution: Parquet is read in batches, and fits that need every value spill to disk.

Install

pip install fast-feature-transform

Wheels cover CPython 3.14, standard and free-threaded (python3.14t), on Linux x86_64 and arm64 (glibc 2.28 or later) and on macOS 26 or later on Apple silicon. Elsewhere, build from source as Development describes.

Usage

from fast_feature_transform import OrdinalEncoder, Pipeline, StandardScaler

pipeline = Pipeline(
    [
        ("country", OrdinalEncoder(columns=["country"])),
        ("numbers", StandardScaler(columns=["age", "income"])),
    ]
).fit(["train.parquet"])

pipeline.transform_dict({"country": "CA", "age": 42, "income": 75000.0})  # serving: one record
pipeline.transform_parquet(["train.parquet"], output_dir="train-features")  # batches: Parquet in, Parquet out
pipeline.transform_memmap(["train.parquet"], output_dir="train-mmap")  # training: one packed file for NumPy

pipeline.save("artifacts/preprocessing")
serving = Pipeline.load("artifacts/preprocessing")

Transform latency

Time to transform one record (a dict in, a dict out) and batches of 100,000 and 1,000,000 rows, after fitting on 100,000 rows. For one record, category_encoders and scikit-learn also pay to build their DataFrame or array and to turn the result back into a dict. A batch goes from one Parquet file to another on both sides. This package reads the file's strings as stored, a dictionary of distinct values plus an index per row; category_encoders and scikit-learn read the file with pyarrow into pandas or NumPy, transform it, and write the result with this package's Parquet settings. Medians of seven rounds on an Apple M4, one thread; lower is better.

At a million rows scikit-learn took 1.14 to 5.3 times as long and category_encoders 1.2 to 30 times. Reading and writing Parquet adds time on both sides, which narrows the gap where the transform is cheap or its output wide; it stays widest where the other library's transform is slow. Matching names don't always mean matching behavior: categories are sorted, output types are fixed at fit, and PolynomialEncoder differs from category_encoders by about 1e-5 at 32 categories and by more than 0.5 from 40 to 128, because category_encoders' basis is the less accurate one.

License

MIT. The wheels also contain Apache Arrow and the other libraries that THIRD_PARTY_NOTICES.txt lists, and code adapted from SciPy, each under its own license.