polars-optbinning 0.2.0

PyO3-Polars OptBinning extension with Rust-side optimization kernels

polars-optbinning

Rust-powered rule binning for Polars DataFrames and expressions.

The main API is BinningTransformer: fit optimized 1D rules for one or more features, keep the fitted rule groups, and reuse them in Polars pipelines.

Installation

Install the published package from PyPI:

pip install polars-optbinning

For local development:

uv sync --group dev
uv run maturin develop --release

Usage

import polars as pl
from polars_optbinning import BinningTransformer, Objective, Stat

df = pl.DataFrame(
    {
        "x1": [1.0, 1.2, 2.0, 2.2, 3.1, 3.4, 4.5, 5.0, 5.2, 6.0],
        "x2": [10.0, 9.8, 9.5, 8.9, 8.1, 7.3, 6.5, 5.9, 5.1, 4.2],
        "y": [0, 0, 0, 1, 0, 1, 1, 1, 1, 1],
    }
)

transformer = BinningTransformer(
    features=[pl.col("x1"), pl.col("x2")],
    target=pl.col("y"),
)

transformer.fit(
    df,
    objective=Objective.iv(),
    mapping=[Stat.woe(), Stat.count()],
    max_bins=4,
)

eager_out = transformer.transform(df)
pipeline_out = df.with_columns(transformer.pl_transform)
print(eager_out)
print(pipeline_out)
print(transformer.summary())

summary() can re-execute the current rules and calculate statistics that are independent of the transform mapping:

stats = transformer.summary(
    extra_stat=[Stat.count(), Stat.avg(), Stat.target_mean()]
)

Fitted transformers reuse their selected training frame. Pass summary(extra_stat=[...], df=new_df) to calculate the statistics on another data set. Parsed and manually constructed transformers require df; pass target="label" as well when requesting target-based statistics. The mapping supplied to fit() or parse() remains the explicit schema produced by transform() and pl_transform; Rule stats not selected by mapping are not automatically expanded into transform columns.

Pairwise 2D binning creates one feature for each feature pair:

pairwise = BinningTransformer(
    features=[pl.col("x1"), pl.col("x2")],
    target=pl.col("y"),
    prefix="pair_",
)

pairwise.fit(
    df,
    objective=Objective.iv(),
    mapping=[Stat.index()],
    max_bins=4,
    mode="pairwise_2d",
    progress=True,
)

out = df.with_columns(pairwise.pl_transform)

For a continuous target regression-style example, see examples/example_linear.py.

Native XGBoost and LightGBM tree models can be parsed into reusable rule groups:

model = BinningTransformer.parse(
    "examples/models/multiclass_xgb.json",
    mapping=[Stat.bin_value()],
    prefix="xgb_",
)

tree_outputs = df.with_columns(model.pl_transform)
raw_margin = df.with_columns(prediction=model.predict())
probability = df.with_columns(prediction=model.predict(output="probability"))
classes = df.with_columns(prediction=model.predict(output="class"))

parse() accepts XGBoost save_model() JSON/UBJ and LightGBM Booster.save_model() text files, paths, and readable file objects. Parsed transformers expose one editable RuleGroup per tree and cannot be fitted again. See examples/example_multiclass_models.py for native model training, serialization, parsing, and Polars prediction comparisons.

Manual rule engines are also supported:

from polars_optbinning import Condition, Rule, RuleGroup

group = RuleGroup(
    "x",
    ["x"],
    [
        Rule(Condition.interval("x", None, 2.0), [Stat.const(0.1)]),
        Rule(Condition.interval("x", 2.0, None), [Stat.const(0.9)]),
    ],
    [Stat.const(0.0)],
)

engine = BinningTransformer(groups=[group])
out = pl.DataFrame({"x": [1.0, 2.5]}).with_columns(engine.pl_transform)

Conditions can be composed as sets across one or more features:

high_risk = (Condition.interval("age", 65.0, None) & Condition.is_in("region", [1.0, 2.0]))
eligible = high_risk | Condition.interval("score", 700.0, None)
excluded = eligible - Condition.is_in("region", [9.0])
assert excluded.features == ["age", "region", "score"]
excluded = excluded.simplify()

group = RuleGroup(
    "decision",
    ["age", "region", "score"],
    [
        Rule(excluded, [Stat.const(1.0)]),
        Rule(Condition.otherwise(), [Stat.const(0.0)]),
    ],
    [Stat.const(0.0)],
)

otherwise is optional, but when present it must be the final rule. Empty groups and unmatched rows without an otherwise rule produce null outputs. Fitted binning groups are ordered by descending fitted count and always include a final otherwise rule.

Rules may be compiled into an explicit binary execution chain:

group.compile()
assert group.execution_kind == "rule_chain"

Perpetual GBDT

Multidimensional gradient boosting is available through the same transformer:

model = BinningTransformer(features=["x1", "x2"], target="y").fit(
    df,
    mode="gbdt",
    objective=Objective.squared_loss(),
    extra_params={
        "budget": 0.5,
        "iteration_limit": 100,
        "categorical_features": ["x2"],
    },
)

raw = df.select(model.predict())
tree_outputs = df.select(model.pl_transform)
importance = model.feature_importance()

Classification uses Objective.log_loss(). Integer class labels are preserved, including non-contiguous labels. extra_params is validated strictly and feature-related settings use feature names rather than column indices.

Tree rules expose Stat.gain(), defined as the cumulative split gain along the root-to-leaf path. feature_importance() instead counts every internal split exactly once and returns feature, gain, gain_normalized, and split_count. Structural rule edits invalidate the original tree importance. See examples/example_perpetual.py for a Titanic survival classification example with categorical and missing-value handling.

The Rust integration currently pins perpetual 3.0.0-rc.2, the first compatible release line that builds on stable Rust; perpetual 2.1.0 requires a nightly-only compiler feature.

Current Scope

• Optimized multi-feature 1D binning via BinningTransformer.fit.
• Typed Objective and Stat selectors.
• Eager transform(df) and expression-friendly pl_transform.
• summary() returning fitted rule statistics as a Polars DataFrame.
• Pairwise 2D rectangular binning via mode="pairwise_2d".
• XGBoost/LightGBM numeric tree parsing, leaf-rule transforms, and model prediction.

ND optimized fitting is intentionally paused while the reusable rule engine interface settles.

Publishing

The release script verifies that pyproject.toml and Cargo.toml use the same version, builds optimized wheel and source distributions, publishes them to PyPI, creates the matching v<version> Gitee tag, and mirrors the files to a Gitee Release.

Token authentication is recommended:

$env:GITEE_TOKEN = "<personal-access-token>"
$env:MATURIN_PYPI_TOKEN = "<pypi-api-token>"
uv run python scripts/gitee_release.py --no-install-project

PYPI_API_TOKEN is accepted as an alias. Account/password authentication is also supported through either MATURIN_USERNAME + MATURIN_PASSWORD or PYPI_USERNAME + PYPI_PASSWORD.

PyPI distributions are immutable. On a repeated run, --skip-existing leaves already published files untouched while allowing wheels for additional platforms to be uploaded. The Gitee Release is updated and same-named attachments are replaced. Existing wheels for other platforms are preserved.

Set SKIP_BUILD=1 to upload files already present in WHEEL_DIR (dist by default), SKIP_PYPI=1 for a Gitee-only mirror, or SKIP_GITEE=1 for a PyPI-only release. All credentials must be stored in the CI service's secret variable store.

Release checklist:

1. Update the version in both pyproject.toml and Cargo.toml.
2. Update CHANGELOG.md.
3. Run Clippy and the Python test suite.
4. Run the release script from the commit that should receive the version tag.

See CONTRIBUTING.md, SECURITY.md, and LICENSE for the project policies.