HydraBoost

Bandit-allocated heterogeneous Newton boosting for tabular data.

Most gradient boosting libraries assume the answer is always an axis-aligned tree. HydraBoost does not. Each boosting round fits one weak learner drawn from a pool of structurally different families, and a bandit allocates rounds to whichever family is actually reducing validation loss. A dataset with smooth or linear structure gets ridge, spline and RBF steps that a tree ensemble would need many splits to approximate; a dense interaction surface gets deep trees. You do not have to know which in advance, and you do not have to tune it to find out.

pip install hydraboost

from hydraboost import HydraBoostClassifier

clf = HydraBoostClassifier().fit(X_train, y_train)
proba = clf.predict_proba(X_test)

print(clf.arm_usage_)
# {'pair': 0.08, 'rbf': 0.14, 'ridge': 0.29, 'spline': 0.09,
#  'tree64': 0.19, 'tree8': 0.21}

That last line is a readout of which learner families kept earning their rounds on your data.

Read it as a diagnostic, not as a structure detector. A family that solves its part of the problem in a few rounds can end up with a small share: on purely linear synthetic data, ridge fits the signal almost immediately and the remaining rounds scatter across arms chasing noise. High share means "kept helping for a long time", which is not the same as "matters most".

How it works

Every round, one learner is fitted from one of six families on the current Newton gradients and Hessians:

Arm selection is epsilon-greedy on an exponential moving average of realised validation improvement, with annealed exploration. Tree arms use their own per-leaf Newton optima; the other arms take a one-step Newton line search so learners on different scales compose correctly.

On small datasets a single validation split is a poor signal for both early stopping and arm feedback, so below small_n rows the estimator pools the split back in and fits several members on rotating splits, averaging them.

API

HydraBoostClassifier and HydraBoostRegressor are standard scikit-learn estimators. They handle categorical columns, missing values, binary and multiclass targets internally, and work inside Pipeline, GridSearchCV, cross_val_score and the rest.

from sklearn.model_selection import cross_val_score
from hydraboost import HydraBoostRegressor

scores = cross_val_score(HydraBoostRegressor(), X, y, cv=5)

Main parameters, all with defaults chosen a priori rather than tuned per dataset:

Benchmarks, and what they do and do not show

HydraBoost was developed against a 16-dataset suite and compared to tuned XGBoost, LightGBM and CatBoost under a shared protocol: 5-fold cross validation, identical folds and design matrices for every model, a 15 percent inner validation split, and test folds never touched during fitting or selection. Each opponent got 8 random hyperparameter configs with early stopping. HydraBoost got one fixed config and no tuning.

On that suite HydraBoost had the best mean metric on 10 of 16 datasets and the best mean rank (1.75, against CatBoost 2.19, LightGBM 2.88, XGBoost 3.19), while fitting 1.5 to 23 times faster than the opponents' average.

These numbers should be read with care, and the honest caveats are these. The algorithm went through four design iterations against that same suite, and iterating against a benchmark inflates significance, so the reported p-values are optimistic. CatBoost's depth grid was capped at 8 for single-threaded compute budget, which handicaps it. Every model ran single-threaded. No untouched large-scale benchmark (OpenML CC-18, Grinsztajn et al.) has been run yet, and until one has, the fair claim is "competitive with tuned GBDTs out of the box on small to mid-size tabular data", not "state of the art".

The full harness and per-fold results are in benchmarks/ so you can rerun or contest any of it. Independent results on datasets I have never seen are the most useful contribution anyone could make here, including negative ones.

Known weak spots from the existing runs: small noisy datasets with many features relative to rows, and problems that are purely dense interaction surfaces, where a dedicated GBDT still wins.

Relation to prior work

HydraBoost combines existing ideas rather than inventing a new one. The specific combination does not appear to be published, but the components are well established and the credit belongs upstream:

• Gradient and Newton boosting: Friedman (2001); Chen and Guestrin (2016)
• Componentwise boosting with base-learner selection per iteration, the direct ancestor of the arm pool: Bühlmann and Yu (2003); the mboost family
• Bandits inside boosting, though for feature selection rather than learner families: Busa-Fekete and Kégl (2009, 2010)
• Heterogeneous base learners across iterations as an alternative to hyperparameter tuning, and cross-validated early stopping for small data: the OGBoost package (2025)
• Pairwise interaction terms in an additive model: the EBM / interpret line of work
• Histogram tree construction: LightGBM (Ke et al., 2017), used directly here

Contributing

Benchmark results on new datasets, especially failures, are the most valuable thing you can send. See CONTRIBUTING.md.

License

MIT