HNBM

Heterogeneous Newton Boosting Machine (HNBM) — a scikit-learn-compatible gradient boosting framework that stochastically mixes heterogeneous base learners at each iteration.

Unlike standard gradient boosting libraries that use a single learner type (typically decision trees), HNBM lets you define a pool of base learners with selection probabilities. At each boosting round, a learner is drawn from that pool and fit to the Newton step (gradient divided by Hessian, weighted by the Hessian).

This is the core framework behind SnapBoost, inspired by SnapBoost: A Heterogeneous Boosting Machine (Parnell et al., NeurIPS 2020).

Table of Contents

• Installation
• Quick Start
• API Reference
  • HNBMClassifier / HNBMRegressor
  • HNBM
  • Loss functions
• Parameters
• Docker
• Development
• Related projects
• License

Installation

From PyPI:

pip install hnbm

From source:

git clone https://github.com/qiancapital-dev/hnbm.git
cd hnbm
pip install .

Requirements: Python ≥ 3.8, NumPy, scikit-learn, tqdm.

Quick Start

Subclass HNBMClassifier or HNBMRegressor and configure your base learner pool before training:

Classification

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor
from hnbm import HNBMClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)


class TreeClassifier(HNBMClassifier):
    def __init__(self, max_depth=5, **kwargs):
        super().__init__(**kwargs)
        self.base_learners_ = [DecisionTreeRegressor(max_depth=max_depth)]
        self.probabilities_ = [1.0]


model = TreeClassifier(
    num_iterations=100,
    learning_rate=0.1,
    random_state=42,
)
model.fit(X_train, y_train)

print("Accuracy:", model.score(X_test, y_test))
print("Probabilities shape:", model.predict_proba(X_test).shape)  # (n_samples, 2)
model.evaluate(X_test, y_test)  # prints log loss

Regression

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor
from hnbm import HNBMRegressor

X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)


class TreeRegressor(HNBMRegressor):
    def __init__(self, max_depth=5, **kwargs):
        super().__init__(**kwargs)
        self.base_learners_ = [DecisionTreeRegressor(max_depth=max_depth)]
        self.probabilities_ = [1.0]


model = TreeRegressor(
    num_iterations=100,
    learning_rate=0.1,
    random_state=42,
)
model.fit(X_train, y_train)

print("R²:", model.score(X_test, y_test))
model.evaluate(X_test, y_test)  # prints RMSE

API Reference

HNBMClassifier / HNBMRegressor

The recommended entry points (similar to XGBClassifier / XGBRegressor). Subclass one of these and set base_learners_ (list of unfitted sklearn regressors) and probabilities_ (list summing to 1) before calling fit.

Methods

HNBM

Legacy base class that accepts a mode parameter ("classification" or "regression"). Prefer HNBMClassifier or HNBMRegressor for new code.

from sklearn.tree import DecisionTreeRegressor
from hnbm import HNBM


class TreeBoost(HNBM):
    def __init__(self, max_depth=5, **kwargs):
        super().__init__(**kwargs)
        self.base_learners_ = [DecisionTreeRegressor(max_depth=max_depth)]
        self.probabilities_ = [1.0]


model = TreeBoost(
    num_iterations=100,
    learning_rate=0.1,
    mode="classification",  # or "regression"
    random_state=42,
)
model.fit(X_train, y_train)

Loss functions

hnbm.losses provides Logistic (classification) and MeanSquaredError (regression), each with a compute_derivatives(y, f) method returning gradient and Hessian vectors.

Parameters

Shared (HNBMClassifier / HNBMRegressor)

The legacy HNBM class also accepts a mode parameter ("classification" or "regression").

Label conventions (classification): accepts 0/1 or -1/+1. Predictions are returned as 0/1.

Docker

docker build -t hnbm .
docker run --rm hnbm

Development

git clone https://github.com/qiancapital-dev/hnbm.git
cd hnbm
pip install -r requirements.txt
pip install -e .

Related projects

• snapboost — a concrete HNBM using decision trees and RFF ridge regressors

License

MIT — See LICENSE for full text.