SmallGBM

Gradient boosting with Bayesian leaf regularization for small data.

Why SmallGBM?

XGBoost and LightGBM are built for scale. They shine on thousands of rows. But when you only have 50, 100, or 500 samples, their default hyperparameters fail — overfitting, instability, unpredictable results.

SmallGBM is designed from the ground up for datasets with fewer than 1000 samples, using Bayesian leaf weight regularization to prevent overfitting on small leaves.

Research

SmallGBM has been evaluated on 16 datasets (synthetic and real-world) with sample sizes from 20 to 1000. Key findings:

• Noise stability: At 20% label noise, SmallGBM outperforms XGBoost and LightGBM
• Prior insensitivity: Performance is nearly invariant to sigma_prior across three orders of magnitude
• Predictable learning curve: Reliable performance begins at n ≈ 40

At 20% label noise, SmallGBM is the best performer. Bayesian regularization keeps it stable when others collapse.

Installation

pip install smallgbm

Quickstart

from smallgbm import SmallGBMClassifier

model = SmallGBMClassifier()
model.fit(X_train, y_train)
proba = model.predict_proba(X_test)

API

SmallGBMClassifier

SmallGBMRegressor

Same parameters, for regression tasks.

from smallgbm import SmallGBMRegressor

model = SmallGBMRegressor()
model.fit(X_train, y_train)
preds = model.predict(X_test)

Citation

@software{emelyanov2026smallgbm,
  author = {Emelyanov, Ilya},
  title = {SmallGBM: Gradient Boosting with Bayesian Leaf Regularization for Small-Sample Tabular Data},
  year = {2026},
  doi = {10.5281/zenodo.21905013},
  url = {https://github.com/nsdmlk/SmallGBM}
}

License

MIT © Emelyanov Ilya 2026

_{Built with ❤️ for the small data community}