Data Science Utils

A toolkit for day-to-day DS tasks such as using custom transformers or estimators.

Found a bug or have a feature request? Open an issue!

Usage

First, import the library:

import pier_ds_utils as ds

Transformers

CustomDiscreteCategorizer

discrete_categorizer = ds.transformer.CustomDiscreteCategorizer(
        column="input_col_name",
        categories=[["my_category_value_1", "my_category_value_2"], ["my_category_value_3"]],
        labels=["label_1", "label_2"],
        default_value="a-default-value",
        output_column="output_col_name",
    )

CustomIntervalCategorizer

interval_categorizer = ds.transformer.CustomIntervalCategorizer(
    column="price",
    intervals=[(6700000, sys.maxsize)],
    labels=["gt_67k"],
    default_value="lt_67k",
    output_column="cat_price",
)

CustomIntervalCategorizerByCategory

interval_categorizer_by_category = ds.transformer.CustomIntervalCategorizerByCategory(
    category_column: "category",
    interval_categorizers: {
        "category_1": CustomIntervalCategorizer(
            column="price",
            intervals=[(6700000, sys.maxsize)],
            labels=["gt_67k"],
            default_value="lt_67k",
            output_column="cat_price",
        ),
        "category_2": CustomIntervalCategorizer(
            column="price",
            intervals=[(0, 1000000)],
            labels=["lt_1M"],
            default_value="gt_1M",
            output_column="cat_price",
        ),
    },
    output_column = "cat_price",
)

CustomMathOperation

math_operation = ds.transformer.CustomMathOperation(
    operation="multiplication",  # also accepts addition, subtraction, division, exponentiation
    column_a="discrete_col",
    column_b="numeric_col",
    output_column="output_col_name",
)

CustomMathOperationByConstant

math_operation_by_constant = ds.transformer.CustomMathOperationByConstant(
    operation="multiplication",  # also accepts addition, subtraction, division, exponentiation
    column="numeric_col",
    constant=1.1,
    output_column="output_col_name",
    invert_order=False,  # set True to compute constant op column instead of column op constant (e.g. constant ** column for exponentiation)
)

LogTransformer

log_transformer = ds.transformer.LogTransformer()

BoundariesTransformer

boundaries_transformer = ds.transformer.BoundariesTransformer(
    lower_bound=0,
    upper_bound=1000000,
)

boundaries_transformer = ds.transformer.BoundariesTransformer(
    lower_bound=0,
    upper_bound=1000000,
    lower_value=10,
    upper_value=1200000
)

CustomBinarizer

Like scikit-learn's Binarizer, but with a configurable condition (">", ">=", "=", "<", "<=") and configurable values. Missing values never meet the condition, so they receive false_value.

binarizer = ds.transformer.CustomBinarizer(
    threshold=100,
    condition=">=",
    true_value="high",
    false_value="low",
)

Estimators

glm_wrapper = ds.estimator.GLMWrapper(...)
predict_proba_selector = ds.estimator.PredictProbaSelector(...)

ClusterLabelMapper wraps a clustering estimator and translates the cluster indexes returned by predict into string labels:

mapper = ds.estimator.ClusterLabelMapper(
    ds.kmodes.KModes(n_clusters=2, random_state=0),
    cluster_map={0: "low_risk", 1: "high_risk"},
)
mapper.fit(df).predict(df)  # array(["low_risk", "high_risk", ...], dtype=object)

Indexes missing from cluster_map raise a ValueError.

Clustering

KModes clusters purely categorical data; KPrototypes clusters mixed numeric and categorical data. Categorical columns of KPrototypes can be given by name (DataFrame input) or by positional index. Missing values are not supported (impute them first); categories not seen during fit are accepted by predict and simply count as mismatches.

KModes

km = ds.kmodes.KModes(n_clusters=3, init="Cao", n_init=5, random_state=0)
labels = km.fit_predict(df[["state", "channel"]])  # all columns are categorical
km.cluster_centroids_, km.cost_

Every column of X is treated as categorical, so select the columns to cluster on before calling fit.

KPrototypes

kp = ds.kmodes.KPrototypes(n_clusters=3, random_state=0,
                           categorical=["state", "channel"])  # the rest is numeric
kp.fit(df)
kp.predict(df_new)

The distance between a row and a centroid is numeric_dissim + gamma * categorical_dissim. Numeric centroids are the cluster means, categorical centroids the cluster modes.

Methods and attributes

• fit(X), fit_predict(X): fit the model (fit_predict also returns the labels); predict(X): assign new rows to the closest cluster. X is a DataFrame or a 2D array.
• labels_: cluster of each training row. cluster_centroids_: array (n_clusters, n_columns) with the centroids in the original column order (modes for categorical, means for numeric columns).
• cost_: total dissimilarity of the best run (lower is better). n_iter_: iterations of the best run. epoch_costs_: cost after each iteration.
• gamma_ (KPrototypes): the weight actually used. categorical_idx_ / numeric_idx_: positions of the categorical and numeric columns. feature_names_in_ / n_features_in_: the columns seen in fit, checked again in predict.

Predictors

predictor = ds.predictor.StaticGLM(...)

Example usage:

from pier_ds_utils.predictor import StaticGLM
import pandas as pd

glm = StaticGLM(
    coefficients_map={"feature1": 0.5, "feature2": 1.5},  # required
    constant=2.0,  # optional
    os_factor=1.0,  # optional
)

df = pd.DataFrame({"feature1": [1, 2], "feature2": [3, 4]})

# The predict is equivalent to:
# y = (0.5 * feature1 + 1.5 * feature2 + constant) * os_factor
print(glm.predict(df))  # Output: [7. 9.]

Installation

pip install pier-ds-utils

# or

poetry add pier-ds-utils

# or

uv add pier-ds-utils

For a specific version:

pip install pier-ds-utils@_version_

# or

poetry add pier-ds-utils@_version_

# or

uv add pier-ds-utils@_version_

Contributing

Contributions are welcome! Please read the contributing guidelines first.