sd-auto-encoder 0.3.0

auto-picks onehot/ordinal/binary encoding per categorical column

SDEncoder

Auto-picks an encoding method per categorical column instead of you manually deciding onehot vs ordinal vs label encoding for each one. Companion to SDScaler, same idea but for categorical data instead of numeric.

Two things in here:

• SDEncoder - for feature columns (X)
• SDTargetEncoder - for your target column (y)

They're kept separate on purpose. Label-style integer codes are fine for a target (a model just needs distinct class ids, order doesn't matter), but doing that on a feature column tricks a lot of models into thinking there's a numeric relationship between categories that don't actually have one. SDEncoder will refuse to touch a Series for this reason, and SDTargetEncoder will refuse a DataFrame.

Install

cd SDEncoder
pip install -e .

Usage

import pandas as pd
from sdencoder import SDEncoder, SDTargetEncoder

df = pd.DataFrame({
    "color": ["red", "blue", "green", "blue"],
    "size": ["S", "M", "L", "M"],
    "is_member": ["yes", "no", "yes", "yes"],
})

encoder = SDEncoder(ordinal_maps={"size": ["S", "M", "L"]})
X_encoded = encoder.fit_transform(df)
print(X_encoded)
print(encoder.summary())

For the target:

y = pd.Series(["approved", "denied", "approved"])
target_encoder = SDTargetEncoder()
y_encoded = target_encoder.fit_transform(y)

Same train/test rule as SDScaler - fit on train, just transform on test:

X_train_enc = encoder.fit_transform(X_train)
X_test_enc = encoder.transform(X_test)

How SDEncoder decides

For each object/category column:

• if you gave it an ordinal_maps entry for that column -> uses your order
• 2 categories -> binary (0/1)
• more than 2 but under onehot_cutoff (default 10) -> onehot
• more than onehot_cutoff -> falls back to ordinal, since onehot on something like a "city" column with 200 values would blow up your feature count for not much benefit

Numeric columns are left completely alone.

ordinal_maps is the important one to use correctly - only pass a real order there for columns that actually have one (like S/M/L, or low/medium/high). Don't use it just to avoid getting a onehot column.

Unseen categories

If .transform() sees a category that wasn't there during .fit(), it becomes NaN for ordinal/binary columns, or an all-zero row for onehot columns. That's intentional - better to notice missing data than have it silently mapped to some made-up number.

ColumnTransformer

If you'd rather use this inside an sklearn Pipeline:

encoder = SDEncoder(ordinal_maps={"size": ["S", "M", "L"]}).fit(df)
ct = encoder.to_column_transformer()

This just builds an sklearn ColumnTransformer using the same decisions SDEncoder already made. Needs scikit-learn installed, only used if you actually call this method.

Files

sdencoder/
  encoder.py   - SDEncoder, for X
  target.py    - SDTargetEncoder, for y
examples/
  example_usage.py
tests/
  test_encoder.py

Todo / ideas

• frequency encoding as another fallback option for high-cardinality columns
• inverse_transform for onehot columns (currently raises NotImplementedError)
• handle missing values (NaN) as their own category instead of erroring later