xgboost-distribution 0.1.0

XGBoost for probabilistic prediction.

xgboost-distribution

XGBoost for probabilistic prediction. Like NGBoost, but faster and in the XGBoost scikit-learn API.

Installation

$ pip install --upgrade xgboost-distribution

Usage

XGBDistribution follows the XGBoost scikit-learn API, except for an additional keyword in the constructor for specifying the distribution. Given some data, we can fit a model:

from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split

from xgboost_distribution import XGBDistribution

data = load_boston()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(X, y)

model = XGBDistribution(
    distribution="normal",
    n_estimators=500
)
model.fit(
    X_train, y_train,
    eval_set=[(X_test, y_test)],
    early_stopping_rounds=10
)

After fitting, we can predict the parameters of the distribution for new data. This will return a namedtuple of numpy arrays for each parameter of the distribution (note that we use scipy naming conventions, see e.g. scipy.stats.norm):

preds = model.predict(X_test)
mean, std = preds.loc, preds.scale

NGBoost performance comparison

XGBDistribution follows the method shown in the NGBoost library, namely using natural gradients to estimate the parameters of the distribution.

Below, we show a performance comparison of the NGBoost NGBRegressor and XGBDistribution models, using the Boston Housing dataset and a normal distribution (similar hyperparameters). We note that while the performance of the two models is essentially identical, XGBDistribution is 50x faster (timed on both fit and predict steps).

Note that the speed-up will decrease with dataset size, as it is ultimately limited by the natural gradient computation (via LAPACK gesv), with 1m rows of data XGBDistribution is still 10x faster than NGBRegressor.

Full XGBoost features

XGBDistribution offers the full set of XGBoost features available in the XGBoost scikit-learn API, allowing, for example, probabilistic prediction with monotonic constraints:

Note

This project has been set up using PyScaffold 4.0.1. For details and usage information on PyScaffold see https://pyscaffold.org/.