HNBM - Heterogeneous Newton Boosting Machine

Heterogeneous Newton Boosting Machine (HNBM) — a scikit-learn-compatible gradient boosting framework that stochastically mixes heterogeneous base learners at each iteration.

Unlike standard gradient boosting libraries that use a single learner type (typically decision trees), HNBM lets you define a pool of base learners with selection probabilities. At each boosting round, a learner is drawn from that pool and fit to the Newton step (gradient divided by Hessian, weighted by the Hessian).

Built-in support includes shallow neural network base learners via NNBoostClassifier / NNBoostRegressor, or you can plug in any scikit-learn-compatible regressor (decision trees, kernel ridge, etc.) by subclassing.

This is the core framework behind SnapBoost, inspired by SnapBoost: A Heterogeneous Boosting Machine (Parnell et al., NeurIPS 2020).

---

Table of Contents

• Installation
• Quick Start
  • Neural networks (NNBoost)
  • Custom base learners (subclassing)
• API Reference
  • NNBoostClassifier / NNBoostRegressor
  • ShallowNNRegressor
  • HNBMClassifier / HNBMRegressor
  • HNBM
  • Loss functions
• Parameters
• Docker
• Development
• Related projects
• License

---

Installation

From PyPI:

pip install hnbm

From source:

git clone https://github.com/qiancapital-dev/hnbm.git
cd hnbm
pip install .

Requirements: Python ≥ 3.8, NumPy, scikit-learn, tqdm.

---

Quick Start

Neural networks (NNBoost)

The fastest way to use HNBM with neural networks is NNBoostClassifier or NNBoostRegressor. Each boosting round randomly selects a single-hidden-layer network from a pool of hidden sizes.

Classification

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from hnbm import NNBoostClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

model = NNBoostClassifier(
    num_iterations=50,
    learning_rate=0.1,
    hidden_layer_sizes=(16, 32, 64),
    learning_rate_nn=0.01,
    max_iter=100,
    random_state=42,
    verbose=False,
)
model.fit(X_train, y_train)

print("Accuracy:", model.score(X_test, y_test))
model.evaluate(X_test, y_test)  # prints log loss

Regression

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from hnbm import NNBoostRegressor

X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

model = NNBoostRegressor(
    num_iterations=50,
    learning_rate=0.1,
    hidden_layer_sizes=(16, 32),
    random_state=42,
    verbose=False,
)
model.fit(X_train, y_train)

print("R²:", model.score(X_test, y_test))
model.evaluate(X_test, y_test)  # prints RMSE

Custom base learners (subclassing)

Subclass HNBMClassifier or HNBMRegressor and configure your own base learner pool before training:

Classification

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor
from hnbm import HNBMClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)


class TreeClassifier(HNBMClassifier):
    def __init__(self, max_depth=5, **kwargs):
        super().__init__(**kwargs)
        self.base_learners_ = [DecisionTreeRegressor(max_depth=max_depth)]
        self.probabilities_ = [1.0]


model = TreeClassifier(
    num_iterations=100,
    learning_rate=0.1,
    random_state=42,
)
model.fit(X_train, y_train)

print("Accuracy:", model.score(X_test, y_test))
print("Probabilities shape:", model.predict_proba(X_test).shape)  # (n_samples, 2)
model.evaluate(X_test, y_test)  # prints log loss

Regression

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor
from hnbm import HNBMRegressor

X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)


class TreeRegressor(HNBMRegressor):
    def __init__(self, max_depth=5, **kwargs):
        super().__init__(**kwargs)
        self.base_learners_ = [DecisionTreeRegressor(max_depth=max_depth)]
        self.probabilities_ = [1.0]


model = TreeRegressor(
    num_iterations=100,
    learning_rate=0.1,
    random_state=42,
)
model.fit(X_train, y_train)

print("R²:", model.score(X_test, y_test))
model.evaluate(X_test, y_test)  # prints RMSE

---

API Reference

NNBoostClassifier / NNBoostRegressor

Ready-to-use HNBM models with a pool of shallow neural network base learners. At each iteration, a network is drawn uniformly from hidden_layer_sizes.

Methods — same as HNBMClassifier / HNBMRegressor (see below).

A legacy NNBoost class is also available with a mode parameter; prefer the task-specific classes for new code.

ShallowNNRegressor

Low-level base learner: a single-hidden-layer network trained with weighted MSE (for Newton step targets and Hessian weights). Supports relu, tanh, and logistic activations.

Use directly in a custom learner pool, or build a pool with make_shallow_nn_pool:

from hnbm import HNBMClassifier, make_shallow_nn_pool

class CustomNNBoost(HNBMClassifier):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.base_learners_, self.probabilities_ = make_shallow_nn_pool(
            hidden_layer_sizes=(16, 32, 64),
            activation="relu",
            max_iter=100,
            random_state=self.random_state,
        )

HNBMClassifier / HNBMRegressor

The recommended entry points (similar to XGBClassifier / XGBRegressor). Subclass one of these and set base_learners_ (list of unfitted sklearn regressors) and probabilities_ (list summing to 1) before calling fit.

Methods

Method	Classifier	Regressor	Description
fit(X, y)	✓	✓	Train the ensemble
predict(X)	✓	✓	Class labels (0/1) or continuous values
predict_proba(X)	✓		Probabilities, shape (n_samples, 2)
decision_function(X)	✓		Raw logits
score(X, y)	✓	✓	Accuracy or R²
evaluate(X, y)	✓	✓	Prints and returns log loss or RMSE

HNBM

Legacy base class that accepts a mode parameter ("classification" or "regression"). Prefer HNBMClassifier or HNBMRegressor for new code.

from sklearn.tree import DecisionTreeRegressor
from hnbm import HNBM


class TreeBoost(HNBM):
    def __init__(self, max_depth=5, **kwargs):
        super().__init__(**kwargs)
        self.base_learners_ = [DecisionTreeRegressor(max_depth=max_depth)]
        self.probabilities_ = [1.0]


model = TreeBoost(
    num_iterations=100,
    learning_rate=0.1,
    mode="classification",  # or "regression"
    random_state=42,
)
model.fit(X_train, y_train)

Loss functions

hnbm.losses provides Logistic (classification) and MeanSquaredError (regression), each with a compute_derivatives(y, f) method returning gradient and Hessian vectors.

---

Parameters

NNBoost (NNBoostClassifier / NNBoostRegressor)

Parameter	Type	Default	Description
num_iterations	int	100	Number of boosting rounds
learning_rate	float	0.1	Boosting shrinkage per learner
hidden_layer_sizes	tuple of int	(16, 32, 64)	Hidden unit counts in the learner pool
activation	str	"relu"	Hidden activation: "relu", "tanh", or "logistic"
alpha	float	1e-4	L2 penalty on network weights
learning_rate_nn	float	0.01	Gradient descent step size per base network
max_iter	int	200	Maximum training epochs per base network
tol	float	1e-5	Early-stopping tolerance on training loss
random_state	int or None	None	Seed for learner selection and weight init
verbose	bool	True	Show tqdm progress bar

Shared (HNBMClassifier / HNBMRegressor)

Parameter	Type	Default	Description
num_iterations	int	100	Number of boosting rounds
learning_rate	float	0.1	Shrinkage per learner
random_state	int or None	None	Seed for learner selection
verbose	bool	True	Show tqdm progress bar

The legacy HNBM class also accepts a mode parameter ("classification" or "regression").

Label conventions (classification): accepts 0/1 or -1/+1. Predictions are returned as 0/1.

---

Docker

docker build -t hnbm .
docker run --rm hnbm

---

Development

git clone https://github.com/qiancapital-dev/hnbm.git
cd hnbm
pip install -r requirements.txt
pip install -e .

---

Related projects

• snapboost — a concrete HNBM using decision trees and RFF ridge regressors
• NNBoost (this package) — a concrete HNBM using shallow neural networks

---

License

MIT — See LICENSE for full text.