Data Science Utils

A toolkit for day-to-day DS tasks such as using custom transformers or estimators.

Found a bug or have a feature request? Open an issue!

Usage

First, import the library:

import pier_ds_utils as ds

Transformers

CustomDiscreteCategorizer

discrete_categorizer = ds.transformer.CustomDiscreteCategorizer(
        column="input_col_name",
        categories=[["my_category_value_1", "my_category_value_2"], ["my_category_value_3"]],
        labels=["label_1", "label_2"],
        default_value="a-default-value",
        output_column="output_col_name",
    )

CustomIntervalCategorizer

interval_categorizer = ds.transformer.CustomIntervalCategorizer(
    column="price",
    intervals=[(6700000, sys.maxsize)],
    labels=["gt_67k"],
    default_value="lt_67k",
    output_column="cat_price",
    output_type="object",  # optional, dtype of the output column (default: "object")
)

CustomIntervalCategorizerByCategory

interval_categorizer_by_category = ds.transformer.CustomIntervalCategorizerByCategory(
    category_column: "category",
    interval_categorizers: {
        "category_1": CustomIntervalCategorizer(
            column="price",
            intervals=[(6700000, sys.maxsize)],
            labels=["gt_67k"],
            default_value="lt_67k",
            output_column="cat_price",
        ),
        "category_2": CustomIntervalCategorizer(
            column="price",
            intervals=[(0, 1000000)],
            labels=["lt_1M"],
            default_value="gt_1M",
            output_column="cat_price",
        ),
    },
    output_column = "cat_price",
)

CustomMathOperation

math_operation = ds.transformer.CustomMathOperation(
    operation="multiplication",  # also accepts addition, subtraction, division, exponentiation
    column_a="discrete_col",
    column_b="numeric_col",
    output_column="output_col_name",
)

CustomMathOperationByConstant

math_operation_by_constant = ds.transformer.CustomMathOperationByConstant(
    operation="multiplication",  # also accepts addition, subtraction, division, exponentiation
    column="numeric_col",
    constant=1.1,
    output_column="output_col_name",
    invert_order=False,  # set True to compute constant op column instead of column op constant (e.g. constant ** column for exponentiation)
)

LogTransformer

log_transformer = ds.transformer.LogTransformer()

BoundariesTransformer

boundaries_transformer = ds.transformer.BoundariesTransformer(
    lower_bound=0,
    upper_bound=1000000,
)

boundaries_transformer = ds.transformer.BoundariesTransformer(
    lower_bound=0,
    upper_bound=1000000,
    lower_value=10,
    upper_value=1200000
)

CustomRounder

Rounds a numeric column up ("up", ceil), down ("down", floor) or to the nearest value ("nearest", halves go to the nearest even number), keeping the chosen number of decimal digits. Negative decimals round to tens, hundreds, etc. It rounds every column it receives, so select the numeric columns beforehand.

rounder = ds.transformer.CustomRounder(
    direction="up",
    decimals=1,
)

CustomBinarizer

Like scikit-learn's Binarizer, but with a configurable condition (">", ">=", "=", "<", "<=") and configurable values. Missing values never meet the condition, so they receive false_value.

binarizer = ds.transformer.CustomBinarizer(
    threshold=100,
    condition=">=",
    true_value="high",
    false_value="low",
)

Estimators

glm_wrapper = ds.estimator.GLMWrapper(...)
predict_proba_selector = ds.estimator.PredictProbaSelector(...)

ClusterLabelMapper wraps a clustering estimator and translates the cluster indexes returned by predict into string labels:

mapper = ds.estimator.ClusterLabelMapper(
    ds.kmodes.KModes(n_clusters=2, random_state=0),
    cluster_map={0: "low_risk", 1: "high_risk"},
)
mapper.fit(df).predict(df)  # array(["low_risk", "high_risk", ...], dtype=object)

Indexes missing from cluster_map raise a ValueError.

Clustering

KModes clusters purely categorical data; KPrototypes clusters mixed numeric and categorical data. Categorical columns of KPrototypes can be given by name (DataFrame input) or by positional index. Missing values are not supported (impute them first); categories not seen during fit are accepted by predict and simply count as mismatches.

KModes

km = ds.kmodes.KModes(n_clusters=3, init="Cao", n_init=5, random_state=0)
labels = km.fit_predict(df[["state", "channel"]])  # all columns are categorical
km.cluster_centroids_, km.cost_

Every column of X is treated as categorical, so select the columns to cluster on before calling fit.

KPrototypes

kp = ds.kmodes.KPrototypes(n_clusters=3, random_state=0,
                           categorical=["state", "channel"])  # the rest is numeric
kp.fit(df)
kp.predict(df_new)

The distance between a row and a centroid is numeric_dissim + gamma * categorical_dissim. Numeric centroids are the cluster means, categorical centroids the cluster modes.

Methods and attributes

• fit(X), fit_predict(X): fit the model (fit_predict also returns the labels); predict(X): assign new rows to the closest cluster. X is a DataFrame or a 2D array.
• labels_: cluster of each training row. cluster_centroids_: array (n_clusters, n_columns) with the centroids in the original column order (modes for categorical, means for numeric columns).
• cost_: total dissimilarity of the best run (lower is better). n_iter_: iterations of the best run. epoch_costs_: cost after each iteration.
• gamma_ (KPrototypes): the weight actually used. categorical_idx_ / numeric_idx_: positions of the categorical and numeric columns. feature_names_in_ / n_features_in_: the columns seen in fit, checked again in predict.

Predictors

predictor = ds.predictor.StaticGLM(...)

Example usage:

from pier_ds_utils.predictor import StaticGLM
import pandas as pd

glm = StaticGLM(
    coefficients_map={"feature1": 0.5, "feature2": 1.5},  # required
    constant=2.0,  # optional
    os_factor=1.0,  # optional
)

df = pd.DataFrame({"feature1": [1, 2], "feature2": [3, 4]})

# The predict is equivalent to:
# y = (0.5 * feature1 + 1.5 * feature2 + constant) * os_factor
print(glm.predict(df))  # Output: [7. 9.]

Installation

pip install pier-ds-utils

# or

poetry add pier-ds-utils

# or

uv add pier-ds-utils

For a specific version:

pip install pier-ds-utils@_version_

# or

poetry add pier-ds-utils@_version_

# or

uv add pier-ds-utils@_version_

Contributing

Contributions are welcome! Please read the contributing guidelines first.