SnapBoost

Heterogeneous Newton Boosting Machine (HNBM) — a gradient boosting framework that mixes decision trees and kernel ridge regressors instead of trees alone. The core HNBM framework is provided by the hnbm package; SnapBoost is a concrete implementation built on top of it.

Unlike XGBoost and LightGBM, which rely exclusively on decision trees as base learners, SnapBoost stochastically selects from a heterogeneous pool of learners at each boosting iteration. This lets the model capture both local, axis-aligned structure (trees) and smooth, global patterns (RBF kernel ridge).

This package is a Python/scikit-learn reimplementation inspired by SnapBoost: A Heterogeneous Boosting Machine (Parnell et al., NeurIPS 2020). See REFERENCES.md for papers, related work, and citation details.

Table of Contents

• Features
• Installation
• Quick Start
• Examples & Results
• API Reference
  • SnapBoostClassifier / SnapBoostRegressor
  • SnapBoost
  • HNBM
• Parameters
• Docker
• Development
• References & Citation
• License

Features

Installation

From PyPI (recommended):

pip install snapboost

From source:

git clone https://github.com/qiancapital/snapboost.git
cd snapboost
pip install .

Requirements: Python ≥ 3.8, NumPy, scikit-learn, tqdm, hnbm ≥ 0.2.2.

Quick Start

Classification

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from snapboost import SnapBoostClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

model = SnapBoostClassifier(
    num_iterations=100,
    learning_rate=0.1,
    random_state=42,
)
model.fit(X_train, y_train)

print("Accuracy:", model.score(X_test, y_test))
print("Probabilities shape:", model.predict_proba(X_test).shape)  # (n_samples, 2)
model.evaluate(X_test, y_test)  # prints log loss

Regression

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from snapboost import SnapBoostRegressor

X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

model = SnapBoostRegressor(
    num_iterations=100,
    learning_rate=0.1,
    random_state=42,
)
model.fit(X_train, y_train)

print("R²:", model.score(X_test, y_test))
model.evaluate(X_test, y_test)  # prints RMSE

Examples & Results

Interactive Jupyter notebooks in static/ walk through classification, regression, and hyperparameter exploration. Each notebook trains SnapBoost and compares it against XGBoost and LightGBM on the same splits.

Run the notebooks locally:

pip install ".[examples]"
jupyter notebook static/

Classification

On the Breast Cancer dataset (250 boosting rounds), SnapBoost achieves the highest test accuracy and fewest misclassifications among the three boosters:

Confusion matrix for SnapBoost on the held-out test set:

Regression

On the Diabetes dataset (100 boosting rounds), SnapBoost improves R² and RMSE over tree-only baselines:

Predicted vs. actual disease progression on the test set:

SnapBoost fitted curve along BMI (other features held at training medians):

Residual distribution:

Parameter exploration

On a synthetic dataset mixing piecewise-linear and sinusoidal structure, the notebook sweeps p_tree, tree depth ranges, and kernel ridge parameters. A mixed ensemble (p_tree=0.8) outperforms trees-only (p_tree=1.0, RMSE 0.174) and ridge-only (p_tree=0.0, RMSE 0.366):

See Parameter_Exploration.ipynb for the full sweeps and baseline comparison tables.

API Reference

SnapBoostClassifier / SnapBoostRegressor

The recommended entry points (similar to XGBClassifier / XGBRegressor). A concrete HNBM that builds an ensemble from:

• Decision trees with depths sampled uniformly from [min_max_depth, max_max_depth]
• One RFF ridge regressor for smooth global fits

At each iteration, a learner is chosen with probability p_tree for trees (split evenly across depths) and 1 - p_tree for the ridge model.

from snapboost import SnapBoostClassifier, SnapBoostRegressor

clf = SnapBoostClassifier(
    num_iterations=100,
    learning_rate=0.1,
    p_tree=0.8,
    min_max_depth=4,
    max_max_depth=8,
    alpha=1.0,
    gamma=1.0,
    random_state=42,
    verbose=True,
)
clf.fit(X, y)

reg = SnapBoostRegressor(num_iterations=100, random_state=42)
reg.fit(X, y)

Methods

SnapBoost

Legacy class that accepts a mode parameter ("classification" or "regression"). Prefer SnapBoostClassifier or SnapBoostRegressor for new code.

from snapboost import SnapBoost

model = SnapBoost(
    num_iterations=100,
    learning_rate=0.1,
    p_tree=0.8,
    min_max_depth=4,
    max_max_depth=8,
    alpha=1.0,
    gamma=1.0,
    mode="classification",  # or "regression"
    random_state=42,
    verbose=True,
)
model.fit(X, y)

Exact kernel ridge variant

For smaller datasets where an exact RBF kernel is preferable to random Fourier features, task-specific exact-kernel estimators are also available:

from snapboost import (
    SnapBoostKernelRidgeClassifier,
    SnapBoostKernelRidgeRegressor,
)

clf = SnapBoostKernelRidgeClassifier(random_state=42)
reg = SnapBoostKernelRidgeRegressor(random_state=42)

Exact kernel ridge has substantially higher memory and runtime costs than the default RFF learner. The old SnapBoost_KernelRidge name remains available for backward compatibility, but new code should use the task-specific classes.

HNBM

The abstract base class for building custom heterogeneous ensembles. Provided by the hnbm package — subclass or configure base_learners_ and probabilities_ before calling fit:

from sklearn.tree import DecisionTreeRegressor
from hnbm import HNBMClassifier, HNBMRegressor

class MyClassifier(HNBMClassifier):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.base_learners_ = [DecisionTreeRegressor(max_depth=5)]
        self.probabilities_ = [1.0]

Parameters

Shared (HNBM / SnapBoostClassifier / SnapBoostRegressor)

The legacy SnapBoost class also accepts a mode parameter ("classification" or "regression").

SnapBoost-specific

Label conventions (classification): accepts any two distinct class labels. Predictions use the original labels, and probability columns follow classes_ order.

Docker

Build and run a container with SnapBoost pre-installed:

docker build -t snapboost .
docker run --rm snapboost

The default command verifies the import:

SnapBoost ready

Development

For normal development against the released HNBM dependency, install SnapBoost in editable mode and run the complete validation suite:

git clone https://github.com/qiancapital/snapboost.git
cd snapboost
python -m pip install -e ".[test]"
python -m pytest -q
python -m compileall -q snapboost tests

The pytest command must finish with all tests passing. To validate SnapBoost against a local sibling checkout of HNBM, install that checkout first:

python -m pip install -e ../hnbm
python -m pip install -e ".[test]"
python -m pytest -q

Run an individual test module or test while developing with:

python -m pytest -q tests/test_snapboost.py
python -m pytest -q tests/test_rff_learner.py
python -m pytest -q tests/test_snapboost.py::test_classifier_preserves_string_labels

The example notebooks require the separate examples dependencies:

python -m pip install -e ".[examples,test]"
jupyter notebook static/

CI runs the full test suite on every push and pull request, and again before a release distribution is built.

Releases are published to PyPI via GitHub Actions when a GitHub release is created.

References & Citation

If you use this package or the HNBM framework in research, please cite the original SnapBoost paper:

Thomas Parnell, Andreea Anghel, Małgorzata Łazuka, Nikolas Ioannou, Sebastian Kurella, Peshal Agarwal, Nikolaos Papandreou, and Haralampos Pozidis. SnapBoost: A Heterogeneous Boosting Machine. Advances in Neural Information Processing Systems, 33, 2020.

@inproceedings{parnell2020snapboost,
  title     = {{SnapBoost}: A Heterogeneous Boosting Machine},
  author    = {Parnell, Thomas and Anghel, Andreea and {\L}azuka, Ma{\l}gorzata and Ioannou, Nikolas and Kurella, Sebastian and Agarwal, Peshal and Papandreou, Nikolaos and Pozidis, Haralampos},
  booktitle = {Advances in Neural Information Processing Systems},
  volume    = {33},
  pages     = {20872--20883},
  year      = {2020},
  eprint    = {2006.09745},
  doi       = {10.48550/arXiv.2006.09745}
}

Links: arXiv:2006.09745 · NeurIPS proceedings · IBM Research

For the full bibliography, related heterogeneous-boosting literature (KTBoost, DeepBoost, etc.), and notes on how this repo relates to the original IBM Snap ML implementation, see REFERENCES.md. Additional BibTeX entries are in CITATION.bib.

License

MIT — See LICENSE for full text.