HNBM - Heterogeneous Newton Boosting Machine

Heterogeneous Newton Boosting Machine (HNBM) — a scikit-learn-compatible gradient boosting framework that stochastically mixes heterogeneous base learners at each iteration.

Unlike standard gradient boosting libraries that use a single learner type (typically decision trees), HNBM lets you define a pool of base learners with selection probabilities. At each boosting round, a learner is drawn from that pool and fit to the Newton step (gradient divided by Hessian, weighted by the Hessian).

Built-in support includes shallow neural network base learners via NNBoostClassifier / NNBoostRegressor. You can also plug in a cloneable scikit-learn regressor whose fit method explicitly accepts sample_weight (for example, decision trees or kernel ridge) by subclassing.

This is the core framework behind SnapBoost, inspired by SnapBoost: A Heterogeneous Boosting Machine (Parnell et al., NeurIPS 2020).

New in 0.3.0: adaptive training

HNBM 0.3.0 adds weighted training, optimized constant base scores, validation history, early stopping with best-ensemble restoration, deterministic row subsampling, greedy learner-family selection, and optional per-round line search. The original stochastic algorithm remains the default with selection_strategy="random".

model = NNBoostRegressor(
    num_iterations=500,
    learning_rate=0.05,
    selection_strategy="greedy",
    line_search=True,
    subsample=0.8,
    early_stopping_rounds=25,
    random_state=42,
)
model.fit(
    X_train,
    y_train,
    sample_weight=train_weights,
    eval_set=(X_validation, y_validation),
)
print(model.best_iteration_, model.history_["validation_loss"])

Additional opt-in extensions include robust and quantile regression objectives, custom per-round metrics, callbacks, parallel greedy candidate fitting, and post-fit model compaction. None changes the default objective or training path.

robust = NNBoostRegressor(
    objective="pseudo_huber",
    objective_parameter=2.0,
    random_state=42,
)
robust.fit(
    X_train,
    y_train,
    eval_metric=lambda y, raw: abs(y - raw).mean(),
    callbacks=[lambda state: state["iteration"] >= 499],
    candidate_n_jobs=2,
)

smaller = robust.compact(min_abs_weight=1e-8)

Table of Contents

• Installation
• Quick Start
  • Neural networks (NNBoost)
  • Custom base learners (subclassing)
• API Reference
  • NNBoostClassifier / NNBoostRegressor
  • ShallowNNRegressor
  • HNBMClassifier / HNBMRegressor
  • HNBM
  • Loss functions
• Parameters
• Docker
• Development
• Related projects
• License

Installation

From PyPI:

pip install hnbm

From source:

git clone https://github.com/qiancapital/hnbm.git
cd hnbm
pip install .

Requirements: Python ≥ 3.8, NumPy, scikit-learn, tqdm.

Quick Start

Neural networks (NNBoost)

The fastest way to use HNBM with neural networks is NNBoostClassifier or NNBoostRegressor. Each boosting round randomly selects a single-hidden-layer network from a pool of hidden sizes.

Classification

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from hnbm import NNBoostClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

model = NNBoostClassifier(
    num_iterations=50,
    learning_rate=0.1,
    hidden_layer_sizes=(16, 32, 64),
    learning_rate_nn=0.01,
    max_iter=100,
    random_state=42,
    verbose=False,
)
model.fit(X_train, y_train)

print("Accuracy:", model.score(X_test, y_test))
model.evaluate(X_test, y_test)  # prints log loss

Regression

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from hnbm import NNBoostRegressor

X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

model = NNBoostRegressor(
    num_iterations=50,
    learning_rate=0.1,
    hidden_layer_sizes=(16, 32),
    random_state=42,
    verbose=False,
)
model.fit(X_train, y_train)

print("R²:", model.score(X_test, y_test))
model.evaluate(X_test, y_test)  # prints RMSE

Custom base learners (subclassing)

Subclass HNBMClassifier or HNBMRegressor and configure your own base learner pool before training:

Classification

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor
from hnbm import HNBMClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)


class TreeClassifier(HNBMClassifier):
    def __init__(self, max_depth=5, **kwargs):
        super().__init__(**kwargs)
        self.base_learners_ = [DecisionTreeRegressor(max_depth=max_depth)]
        self.probabilities_ = [1.0]


model = TreeClassifier(
    num_iterations=100,
    learning_rate=0.1,
    random_state=42,
)
model.fit(X_train, y_train)

print("Accuracy:", model.score(X_test, y_test))
print("Probabilities shape:", model.predict_proba(X_test).shape)  # (n_samples, 2)
model.evaluate(X_test, y_test)  # prints log loss

Regression

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor
from hnbm import HNBMRegressor

X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)


class TreeRegressor(HNBMRegressor):
    def __init__(self, max_depth=5, **kwargs):
        super().__init__(**kwargs)
        self.base_learners_ = [DecisionTreeRegressor(max_depth=max_depth)]
        self.probabilities_ = [1.0]


model = TreeRegressor(
    num_iterations=100,
    learning_rate=0.1,
    random_state=42,
)
model.fit(X_train, y_train)

print("R²:", model.score(X_test, y_test))
model.evaluate(X_test, y_test)  # prints RMSE

API Reference

NNBoostClassifier / NNBoostRegressor

Ready-to-use HNBM models with a pool of shallow neural network base learners. At each iteration, a network is drawn uniformly from hidden_layer_sizes.

Methods — same as HNBMClassifier / HNBMRegressor (see below).

A legacy NNBoost class is also available with a mode parameter; prefer the task-specific classes for new code.

ShallowNNRegressor

Low-level base learner: a single-hidden-layer network trained with weighted MSE (for Newton step targets and Hessian weights). Supports relu, tanh, and logistic activations.

Use directly in a custom learner pool, or build a pool with make_shallow_nn_pool:

from hnbm import HNBMClassifier, make_shallow_nn_pool

class CustomNNBoost(HNBMClassifier):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.base_learners_, self.probabilities_ = make_shallow_nn_pool(
            hidden_layer_sizes=(16, 32, 64),
            activation="relu",
            max_iter=100,
            random_state=self.random_state,
        )

HNBMClassifier / HNBMRegressor

The recommended entry points (similar to XGBClassifier / XGBRegressor). Subclass one of these and set base_learners_ (a list of unfitted, cloneable regressors whose fit methods explicitly accept sample_weight) and probabilities_ (a list of finite, nonnegative values summing to 1) before calling fit.

Methods

After fitting, n_iter_ contains the number of completed boosting rounds. The inner epoch count for each fitted neural-network learner remains available on that learner's own n_iter_ attribute.

New fitted attributes in 0.3.0 include base_score_, learner_weights_, history_, and best_iteration_. Validation loss is recorded only when eval_set=(X_validation, y_validation) is supplied. Early stopping requires an evaluation set and restores the best ensemble before returning.

HNBM

Legacy base class that accepts a mode parameter ("classification" or "regression"). Prefer HNBMClassifier or HNBMRegressor for new code.

from sklearn.tree import DecisionTreeRegressor
from hnbm import HNBM


class TreeBoost(HNBM):
    def __init__(self, max_depth=5, **kwargs):
        super().__init__(**kwargs)
        self.base_learners_ = [DecisionTreeRegressor(max_depth=max_depth)]
        self.probabilities_ = [1.0]


model = TreeBoost(
    num_iterations=100,
    learning_rate=0.1,
    mode="classification",  # or "regression"
    random_state=42,
)
model.fit(X_train, y_train)

Loss functions

hnbm.losses provides Logistic (classification) and MeanSquaredError (regression), each with a compute_derivatives(y, f) method returning gradient and Hessian vectors.

Parameters

NNBoost (NNBoostClassifier / NNBoostRegressor)

Shared (HNBMClassifier / HNBMRegressor)

The legacy HNBM class also accepts a mode parameter ("classification" or "regression").

Label conventions (classification): accepts any two distinct class labels. Predictions use the original labels, and probability columns follow classes_ order.

Docker

docker build -t hnbm .
docker run --rm hnbm

Development

Create an environment, install HNBM in editable mode with its test dependencies, and run the complete validation suite:

git clone https://github.com/qiancapital/hnbm.git
cd hnbm
python -m pip install -e ".[test]"
python -m pytest -q
python -m compileall -q hnbm tests

The pytest command must finish with all tests passing. To run an individual test module or a single test while developing:

python -m pytest -q tests/test_hnbm.py
python -m pytest -q tests/test_nn_learner.py
python -m pytest -q tests/test_hnbm.py::test_classifier_preserves_arbitrary_binary_labels

CI runs the full test suite on every push and pull request, and again before a release distribution is built.

Related projects

• snapboost — a concrete HNBM using decision trees and RFF ridge regressors
• NNBoost (this package) — a concrete HNBM using shallow neural networks

License

MIT — See LICENSE for full text.