RXGB — Random eXtreme Gradient Boosting

RXGB is a robust ensemble framework that combines Random Forest-style bagging with depth-constrained XGBoost learners and a Dynamic Attention mechanism. It is specifically designed to improve prediction robustness and performance on challenging tabular datasets, including noisy, imbalanced, and highly complex data.

Overview

RXGB wraps multiple independent XGBoost models in a bagging-style ensemble. Each base learner is trained using a randomized sample of rows and features, and the ensemble combines their predictions.

The design brings together three powerful concepts:

• Boosting (XGBoost): Builds trees sequentially within each base learner to capture complex relationships and reduce bias.
• Bagging (Random Forest): Trains independent base learners on different samples and feature subsets to drastically reduce prediction variance.
• Attention (Dynamic Weighting): Uses a meta-learner to dynamically weight the predictions of base learners, preventing weak models from ruining the final prediction on noisy data.

Why RXGB and not standard XGBoost?

Standard XGBoost is highly sensitive to noise and can easily overfit on small, complex datasets. RXGB solves this by placing XGBoost inside a bagging architecture. By dynamically weighting the predictions of multiple independent XGBoost models, RXGB simultaneously reduces bias and variance. Our benchmarks show that RXGB consistently outperforms standard XGBoost, LightGBM, CatBoost, and Random Forest on difficult datasets.

How It Works

RXGB uses two levels of model construction:

1. Row and Feature subsampling: Each base learner trains on a randomized sample of rows (max_samples) and features (max_features).
2. Independent XGBoost learners: Each base learner fits its own boosted-tree model.
3. Dynamic Attention (Default): Instead of simple averaging, a meta-learner (Logistic Regression for classification, Ridge for regression) learns the optimal weight for each base learner based on the input features.

Unlike a single XGBoost model, RXGB combines predictions from multiple independently trained models with dynamic weighting. This is what prevents overfitting on noisy data and stabilizes performance on imbalanced classes.

Installation

Install RXGB from PyPI:

pip install rxgb

Quick Start

from rxgb import RXGBClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# 1. Create a challenging synthetic dataset (noisy)
X, y = make_classification(
    n_samples=2000, n_features=20, n_informative=15,
    flip_y=0.15, random_state=42
)

# 2. Split the data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# 3. Initialize and train RXGB (Attention is enabled by default)
clf = RXGBClassifier(
    n_estimators=25,
    max_depth=4,
    dynamic_weighting=True, # Set to False for classic Bagging
    random_state=42
)
clf.fit(X_train, y_train)

# 4. Predict and evaluate
predictions = clf.predict(X_test)
print(f"Accuracy: {accuracy_score(y_test, predictions):.4f}")

Performance Benchmarks

We evaluated RXGB-Attention against industry-standard models across highly challenging datasets (including extreme noise, high overlap, and high-dimensional data).

Note: The benchmark reflects performance on challenging, noisy, and overlapping datasets. RXGB-Attention achieves superior F1-scores without any hyperparameter tuning, outperforming heavily optimized baselines like XGBoost and CatBoost on difficult real-world data, while maintaining a competitive training time.

Parameters

Documentation and Source Code

• GitHub repository: https://github.com/Kia-ML/rxgb
• PyPI package: https://pypi.org/project/rxgb/
• Issue tracker: https://github.com/Kia-ML/rxgb/issues

The repository contains the full source code, automated tests (PyTest), and reproducible benchmark scripts.

Citation

If you use RXGB in your research, please cite the work:

@article{nouri2026rxgb,
  title   = {RXGB: A Bagging Ensemble of Shallow XGBoost Learners for Classification},
  author  = {Nouri, Kiarash},
  journal = {Journal of Machine Learning Research (Under Review)},
  year    = {2026}
}

License

RXGB is distributed under the MIT License. See the LICENSE file for the full license text.