HNBM - Heterogeneous Newton Boosting Machine

Heterogeneous Newton Boosting Machine (HNBM) — a scikit-learn-compatible gradient boosting framework that stochastically mixes heterogeneous base learners at each iteration.

Unlike standard gradient boosting libraries that use a single learner type (typically decision trees), HNBM lets you define a pool of base learners with selection probabilities. At each boosting round, a learner is drawn from that pool and fit to the Newton step (gradient divided by Hessian, weighted by the Hessian).

Built-in support includes shallow neural network base learners via NNBoostClassifier / NNBoostRegressor. You can also plug in a cloneable scikit-learn regressor whose fit method explicitly accepts sample_weight (for example, decision trees or kernel ridge) by subclassing.

This is the core framework behind SnapBoost, inspired by SnapBoost: A Heterogeneous Boosting Machine (Parnell et al., NeurIPS 2020).

New in 1.2

HNBM 1.2 adds native multiclass classification with softmax Newton boosting. Binary logistic classification is unchanged: two classes still use a scalar logit. Multiclass models return decision_function of shape (n_samples, n_classes) and store one scalar learner per class each round.

from sklearn.datasets import load_iris
from hnbm import NNBoostClassifier

X, y = load_iris(return_X_y=True)
model = NNBoostClassifier(num_iterations=50, random_state=42, verbose=False)
model.fit(X, y)
print(model.n_classes_, model.predict_proba(X).shape)  # 3, (n_samples, 3)

New in 1.1

HNBM 1.1 passes original labels to eval_metric (including string class names), accepts eval_sample_weight for validation loss and early stopping, and adds staged_predict / staged_predict_proba / staged_decision_function plus permutation_importance.

staged = list(model.staged_predict(X))
importance = model.permutation_importance(X, y, n_repeats=5, random_state=42)

New in 1.0

HNBM 1.0 freezes HNBMClassifier / HNBMRegressor, reports sklearn estimator tags (dense inputs, no native missing values), and delays parameter validation until fit. Constructing HNBM(mode=...) or NNBoost(mode=...) directly is deprecated.

New in 0.3.0: adaptive training

HNBM 0.3.0 adds weighted training, optimized constant base scores, validation history, early stopping with best-ensemble restoration, deterministic row subsampling, greedy learner-family selection, and optional per-round line search. The original stochastic algorithm remains the default with selection_strategy="random".

model = NNBoostRegressor(
    num_iterations=500,
    learning_rate=0.05,
    selection_strategy="greedy",
    line_search=True,
    subsample=0.8,
    early_stopping_rounds=25,
    random_state=42,
)
model.fit(
    X_train,
    y_train,
    sample_weight=train_weights,
    eval_set=(X_validation, y_validation),
)
print(model.best_iteration_, model.history_["validation_loss"])

Additional opt-in extensions include robust and quantile regression objectives, custom per-round metrics, callbacks, parallel greedy candidate fitting, and post-fit model compaction. None changes the default objective or training path.

robust = NNBoostRegressor(
    objective="pseudo_huber",
    objective_parameter=2.0,
    random_state=42,
)
robust.fit(
    X_train,
    y_train,
    eval_metric=lambda y, raw: abs(y - raw).mean(),
    callbacks=[lambda state: state["iteration"] >= 499],
    candidate_n_jobs=2,
)

smaller = robust.compact(min_abs_weight=1e-8)

Table of Contents

• New in 1.2
• Mathematical Overview
• Installation
• Quick Start
  • Neural networks (NNBoost)
  • Custom base learners (subclassing)
• API Reference
  • NNBoostClassifier / NNBoostRegressor
  • ShallowNNRegressor
  • HNBMClassifier / HNBMRegressor
  • HNBM
  • Loss functions
• Parameters
• Docker
• Development
• Related projects
• License

Mathematical Overview

HNBM constructs an additive predictor from a probability-weighted pool of possibly different hypothesis classes:

$$ F_M(x)=F_0+\sum_{m=1}^{M}\eta_m f_m(x), \qquad f_m\in\mathcal H_{K_m}, \qquad K_m\sim\mathrm{Categorical}(p_1,\ldots,p_K). $$

Here $F_0$ is a constant initial score, $\eta_m$ is a boosting step size, and $\mathcal H_{K_m}$ may contain trees, kernels, neural networks, linear models, or any cloneable weighted regressor. The probabilities satisfy $p_k\geq0$ and $\sum_kp_k=1$.

At boosting round $m$, HNBM differentiates the loss at the current raw prediction:

$$ g_i=\left.\frac{\partial\ell(y_i,F)}{\partial F}\right|{F=F{m-1}(x_i)}, \qquad h_i=\left.\frac{\partial^2\ell(y_i,F)}{\partial F^2}\right|{F=F{m-1}(x_i)}. $$

Completing the square in the second-order Taylor approximation shows that the chosen learner should solve the weighted regression problem

$$ r_i=-\frac{g_i}{h_i}, \qquad f_m\approx\arg\min_{f\in\mathcal H_{K_m}} \sum_{i=1}^{n}w_i h_i\bigl(r_i-f(x_i)\bigr)^2, $$

followed by

$$ F_m(x)=F_{m-1}(x)+\eta_m f_m(x). $$

Thus $-g_i/h_i$ is the Newton working response and $w_i h_i$ is its effective sample weight. For squared-error regression, $g_i=2(F-y_i)$ and $h_i=2$, so $r_i=y_i-F$: ordinary residual boosting is recovered. For binary logistic classification with $y_i\in{-1,+1}$,

$$ \ell(y,F)=\log(1+e^{-yF}),\quad g=-y\sigma(-yF),\quad h=\sigma(yF)\sigma(-yF), $$

and $P(y=+1\mid x)=\sigma(F_M(x))$. Multiclass targets instead use a $K$-vector score $F(x)\in\mathbb{R}^K$ with

$$ p=\mathrm{softmax}(F),\quad \ell=-\log p_{y},\quad g_k=p_k-\mathbf{1}_{{k=y}},\quad h_k=p_k(1-p_k), $$

$$ r_k=-\frac{g_k}{h_k} =\frac{\mathbf{1}{{k=y}}-p_k}{p_k(1-p_k)}, \qquad F{0,k}=\log\widehat p_k. $$

Each round fits $K$ scalar learners from the same family and $P(y=k\mid x)=\mathrm{softmax}(F_M(x))_k$. See MATH.md for the full Hessian, the diagonal approximation, and the stable log-sum-exp form.

The included NNBoost realization uses a uniform pool of one-hidden-layer networks. For hidden width $q$ its correction is

$$ f(x)=v^\top a(W^\top\widetilde x+b)+c, $$

and the network is trained against the Newton responses using Hessian weights and L2 regularization. Different widths define different subclasses $\mathcal H_q$; the default random strategy samples one width per round, while the optional greedy strategy fits every eligible width and keeps the update with the lowest loss.

Traditional gradient boosting fits $-g_i$ and usually uses a single learner class. XGBoost uses a related Newton expansion but specializes it to regularized trees with analytic leaf weights and split gains. HNBM separates the Newton optimization interface from the learner type, allowing different inductive biases to coexist in one ensemble.

See MATH.md for the complete derivation, objective formulas, stochastic and greedy HNBM algorithms, NNBoost training equations, convergence interpretation, and detailed comparisons with gradient boosting, Newton tree boosting, SnapBoost, and XGBoost.

Installation

From PyPI:

pip install hnbm

From source:

git clone https://github.com/qiancapital/hnbm.git
cd hnbm
pip install .

Requirements: Python ≥ 3.8, NumPy, scikit-learn, tqdm.

Quick Start

Neural networks (NNBoost)

The fastest way to use HNBM with neural networks is NNBoostClassifier or NNBoostRegressor. Each boosting round randomly selects a single-hidden-layer network from a pool of hidden sizes.

Classification

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from hnbm import NNBoostClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

model = NNBoostClassifier(
    num_iterations=50,
    learning_rate=0.1,
    hidden_layer_sizes=(16, 32, 64),
    learning_rate_nn=0.01,
    max_iter=100,
    random_state=42,
    verbose=False,
)
model.fit(X_train, y_train)

print("Accuracy:", model.score(X_test, y_test))
model.evaluate(X_test, y_test)  # prints log loss

Multiclass labels use softmax Newton boosting. predict_proba and decision_function then have one column per class:

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from hnbm import NNBoostClassifier

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

model = NNBoostClassifier(num_iterations=50, random_state=42, verbose=False)
model.fit(X_train, y_train)
print("Classes:", model.n_classes_)
print("Probabilities shape:", model.predict_proba(X_test).shape)  # (n_samples, 3)

Regression

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from hnbm import NNBoostRegressor

X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

model = NNBoostRegressor(
    num_iterations=50,
    learning_rate=0.1,
    hidden_layer_sizes=(16, 32),
    random_state=42,
    verbose=False,
)
model.fit(X_train, y_train)

print("R²:", model.score(X_test, y_test))
model.evaluate(X_test, y_test)  # prints RMSE

Custom base learners (subclassing)

Subclass HNBMClassifier or HNBMRegressor and configure your own base learner pool before training:

Classification

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor
from hnbm import HNBMClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)


class TreeClassifier(HNBMClassifier):
    def __init__(self, max_depth=5, **kwargs):
        super().__init__(**kwargs)
        self.base_learners_ = [DecisionTreeRegressor(max_depth=max_depth)]
        self.probabilities_ = [1.0]


model = TreeClassifier(
    num_iterations=100,
    learning_rate=0.1,
    random_state=42,
)
model.fit(X_train, y_train)

print("Accuracy:", model.score(X_test, y_test))
print("Probabilities shape:", model.predict_proba(X_test).shape)  # (n_samples, n_classes)
model.evaluate(X_test, y_test)  # prints log loss

Regression

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor
from hnbm import HNBMRegressor

X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)


class TreeRegressor(HNBMRegressor):
    def __init__(self, max_depth=5, **kwargs):
        super().__init__(**kwargs)
        self.base_learners_ = [DecisionTreeRegressor(max_depth=max_depth)]
        self.probabilities_ = [1.0]


model = TreeRegressor(
    num_iterations=100,
    learning_rate=0.1,
    random_state=42,
)
model.fit(X_train, y_train)

print("R²:", model.score(X_test, y_test))
model.evaluate(X_test, y_test)  # prints RMSE

API Reference

NNBoostClassifier / NNBoostRegressor

Ready-to-use HNBM models with a pool of shallow neural network base learners. At each iteration, a network is drawn uniformly from hidden_layer_sizes.

Methods — same as HNBMClassifier / HNBMRegressor (see below).

A legacy NNBoost class is also available with a mode parameter; prefer the task-specific classes for new code.

ShallowNNRegressor

Low-level base learner: a single-hidden-layer network trained with weighted MSE (for Newton step targets and Hessian weights). Supports relu, tanh, and logistic activations.

Use directly in a custom learner pool, or build a pool with make_shallow_nn_pool:

from hnbm import HNBMClassifier, make_shallow_nn_pool

class CustomNNBoost(HNBMClassifier):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.base_learners_, self.probabilities_ = make_shallow_nn_pool(
            hidden_layer_sizes=(16, 32, 64),
            activation="relu",
            max_iter=100,
            random_state=self.random_state,
        )

HNBMClassifier / HNBMRegressor

The recommended entry points (similar to XGBClassifier / XGBRegressor). Subclass one of these and set base_learners_ (a list of unfitted, cloneable regressors whose fit methods explicitly accept sample_weight) and probabilities_ (a list of finite, nonnegative values summing to 1) before calling fit.

Methods

After fitting, n_iter_ contains the number of completed boosting rounds. Classifiers also expose classes_ and n_classes_. The inner epoch count for each fitted neural-network learner remains available on that learner's own n_iter_ attribute. eval_metric receives original labels (including string class names). eval_sample_weight is used for validation loss and early stopping when an eval_set is provided.

New fitted attributes in 0.3.0 include base_score_, learner_weights_, history_, and best_iteration_. Validation loss is recorded only when eval_set=(X_validation, y_validation) is supplied. Early stopping requires an evaluation set and restores the best ensemble before returning.

HNBM

Legacy base class that accepts a mode parameter ("classification" or "regression"). Prefer HNBMClassifier or HNBMRegressor for new code.

from sklearn.tree import DecisionTreeRegressor
from hnbm import HNBM


class TreeBoost(HNBM):
    def __init__(self, max_depth=5, **kwargs):
        super().__init__(**kwargs)
        self.base_learners_ = [DecisionTreeRegressor(max_depth=max_depth)]
        self.probabilities_ = [1.0]


model = TreeBoost(
    num_iterations=100,
    learning_rate=0.1,
    mode="classification",  # or "regression"
    random_state=42,
)
model.fit(X_train, y_train)

Loss functions

hnbm.losses provides Logistic (binary classification), Softmax (multiclass classification), MeanSquaredError, PseudoHuber, and Quantile (regression). Each exposes compute_derivatives and compute_loss, returning gradient and Hessian arrays for the Newton step. Logistic, Softmax, and MeanSquaredError take (y, f); PseudoHuber and Quantile take (y, f, objective_parameter).

Parameters

NNBoost (NNBoostClassifier / NNBoostRegressor)

Shared (HNBMClassifier / HNBMRegressor)

For classification, both objective="auto" and objective="log_loss" select logistic loss for binary targets and softmax for multiclass targets.

Choosing the Pseudo-Huber delta. The Newton working response for pseudo-Huber grows like residual³ / delta², so the default delta=1.0 diverges on targets that are not roughly unit-scale. Standardize y, or set objective_parameter to about the residual scale. This is the same consideration as huber_slope in XGBoost's reg:pseudohubererror.

The legacy HNBM class also accepts a mode parameter ("classification" or "regression").

Label conventions (classification): binary and multiclass labels are accepted. Predictions use the original labels, and probability columns follow classes_ order. Binary models keep a scalar decision_function; multiclass models return one column per class. A target with a single class raises ValueError.

fit(X, y, sample_weight=None, eval_set=None, *, eval_sample_weight=None) accepts non-negative observation weights, one validation pair, and optional validation weights. eval_metric(y, raw) receives the original labels, not an internal encoding. After fit, staged_predict (and classifier staged_predict_proba / staged_decision_function) yield the ensemble after each round. permutation_importance(X, y) is the recommended feature-importance API.

Docker

docker build -t hnbm .
docker run --rm hnbm

Development

Create an environment, install HNBM in editable mode with its test dependencies, and run the complete validation suite:

git clone https://github.com/qiancapital/hnbm.git
cd hnbm
python -m pip install -e ".[test]"
python -m pytest -q
python -m compileall -q hnbm tests

The pytest command must finish with all tests passing. To run an individual test module or a single test while developing:

python -m pytest -q tests/test_hnbm.py
python -m pytest -q tests/test_nn_learner.py
python -m pytest -q tests/test_hnbm.py::test_classifier_preserves_arbitrary_binary_labels

CI runs the full test suite on every push and pull request, and again before a release distribution is built.

Related projects

• snapboost — a concrete HNBM using decision trees and RFF ridge regressors
• NNBoost (this package) — a concrete HNBM using shallow neural networks

License

MIT — See LICENSE for full text.