GBRF — Gradient Boosted Random Forest

A novel hybrid machine learning algorithm combining Random Forest and Gradient Boosting. At each boosting iteration, a full Random Forest ensemble learns the pseudo-residuals, providing superior variance reduction compared to standard Gradient Boosting.

What Makes GBRF Different?

The Innovation: Instead of a single decision tree learning residuals at each boosting step (standard GB), GBRF uses a Random Forest ensemble per iteration. This gives you:

• Better generalization than Gradient Boosting alone
• Faster convergence than Random Forest alone
• Built-in early stopping via out-of-bag monitoring
• Aggregated feature importance across all boosting rounds

Installation

pip install hybridgbrf

Requirements

• Python >= 3.8
• numpy >= 1.21.0
• scikit-learn >= 1.0.0
• pandas >= 1.3.0
• matplotlib >= 3.4.0
• joblib >= 1.1.0

Quick Start

Regression

from hybridgbrf import GRFRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split

# Create data
X, y = make_regression(n_samples=1000, n_features=20, noise=0.1, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Train GBRF
model = GRFRegressor(
    n_iterations=50,
    n_estimators_per_iteration=10,
    learning_rate=0.1,
    max_depth=3,
    early_stopping_rounds=5,
    random_state=42
)

model.fit(X_train, y_train)

# Evaluate
print(f"R^2 Score: {model.score(X_test, y_test):.4f}")
print(f"MSE: {model.mse(X_test, y_test):.4f}")

# Feature importance
model.plot_feature_importances(top_n=10, save_path="importance.png")

# Training history
model.plot_training_history(save_path="history.png")

Classification

from hybridgbrf import GRFClassifier
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=1000, n_features=20, n_classes=2, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

model = GRFClassifier(
    n_iterations=50,
    n_estimators_per_iteration=10,
    learning_rate=0.1,
    early_stopping_rounds=5,
    random_state=42
)

model.fit(X_train, y_train)

# Predict
y_pred = model.predict(X_test)
y_proba = model.predict_proba(X_test)

# Evaluate
print(f"Accuracy: {model.score(X_test, y_test):.4f}")
print(model.classification_report(X_test, y_test))

API Reference

GRFRegressor

Methods

How It Works

Standard Gradient Boosting:          GBRF (Our Hybrid):
+--------------------------------+   +--------------------------------+
|  Iteration 1                   |   |  Iteration 1                   |
|  Tree -> Residual Prediction   |   |  RF Ensemble -> Residuals      |
+--------------------------------+   +--------------------------------+
              |                                    |
              v                                    v
+--------------------------------+   +--------------------------------+
|  Iteration 2                   |   |  Iteration 2                   |
|  Tree -> Residual Prediction   |   |  RF Ensemble -> Residuals      |
+--------------------------------+   +--------------------------------+
              |                                    |
              v                                    v
       ... continues ...                    ... continues ...

Key Difference: Single Tree vs Random Forest per iteration

Performance Comparison

from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from hybridgbrf import GRFRegressor
from sklearn.datasets import fetch_california_housing

X, y = fetch_california_housing(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Random Forest
rf = RandomForestRegressor(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)
print(f"RF R^2: {rf.score(X_test, y_test):.4f}")

# Gradient Boosting
gb = GradientBoostingRegressor(n_estimators=100, random_state=42)
gb.fit(X_train, y_train)
print(f"GB R^2: {gb.score(X_test, y_test):.4f}")

# GBRF (Ours)
hgbrf = GRFRegressor(n_iterations=50, n_estimators_per_iteration=10, random_state=42)
hgbrf.fit(X_train, y_train)
print(f"GBRF R^2: {hgbrf.score(X_test, y_test):.4f}")

Saving and Loading

# Save
model.save_model("my_gbrf_model.pkl")

# Load
from gbrf import GRFRegressor
model = GRFRegressor.load_model("my_gbrf_model.pkl")

License

MIT License -- see LICENSE for details.

Built with ❤️ by Abhishek Singh