WRFO: Weighted Random Forest Optimization

WRFO (Weighted Random Forest Optimization) is a machine learning algorithm that enhances Random Forest classifiers through tree weighting, accuracy and diversity-preserving pruning using Particle Swarm Optimization (PSO). By optimizing tree weights based on a multi-objective function combining accuracy, diversity, and feature entropy, WRFO achieves improved performance while using fewer trees.

Paper: "Towards Better Random Forests with Tree Weighting, Accuracy and Diversity-Preserving Pruning" (Expert Systems with Applications, under revision)

Key Features

• Multi-Objective Optimization: Balances accuracy, ensemble diversity, and feature entropy
• Adaptive Tree Selection: Automatically identifies and weights the most valuable trees
• Scikit-learn Compatible: Drop-in replacement for sklearn's RandomForestClassifier
• Parallel Processing: Efficient computation using joblib parallelization
• Research-Validated: under revision in Expert Systems with Applications

Installation

From source

git clone https://github.com/wajdidhifli/WRFO.git
cd WRFO
pip install -r requirements.txt
pip install -e .

Requirements

• Python >= 3.7
• numpy >= 1.20.0
• pandas >= 1.3.0
• scikit-learn >= 1.0.0
• pyswarm >= 0.6
• scipy >= 1.7.0
• joblib >= 1.0.0
• tqdm >= 4.62.0

Quick Start

from wrfo import WRFOClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

# Load data
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

# Train WRFO
clf = WRFOClassifier(n_estimators=100, random_state=42)
clf.fit(X_train, y_train)

# Predict
y_pred = clf.predict(X_test)
print(f"Accuracy: {clf.score(X_test, y_test):.3f}")

Usage

Basic Classification

from wrfo import WRFOClassifier

# Initialize with default parameters
wrfo = WRFOClassifier(
    n_estimators=100,      # Number of trees
    swarm_size=10,         # PSO swarm size
    max_iter=10,           # PSO iterations
    random_state=42        # For reproducibility
)

# Fit and predict
wrfo.fit(X_train, y_train)
predictions = wrfo.predict(X_test)

Advanced Configuration

wrfo = WRFOClassifier(
    n_estimators=100,
    swarm_size=10,
    max_iter=10,
    accuracy_weight=0.6,    # Weight for accuracy in objective
    diversity_weight=0.4,   # Weight for diversity in objective
    entropy_weight=0.1,     # Weight for entropy in objective
    val_split=0.2,          # Validation split for PSO optimization
    random_state=42,
    n_jobs=-1,              # Use all CPU cores
    verbose=True            # Print progress
)

Access Optimized Weights

# After fitting
print(f"Optimized weights: {wrfo.weights_}")
print(f"Trees selected: {sum(wrfo.weights_ > 0)}/{wrfo.n_estimators}")
print(f"Diversity matrix shape: {wrfo.divmat_.shape}")

Examples

See the examples/ directory for complete working examples:

• iris_classification.py: Cross-validation example on Iris dataset
• custom_dataset_example.py: Template for using WRFO with custom datasets

Run an example:

cd examples
python iris_classification.py

How It Works

WRFO improves upon standard Random Forest through three key steps:

1. Train Base Ensemble: Creates a Random Forest with n_estimators trees
2. Compute Diversity Matrix: Calculates pairwise Cohen's kappa diversity between all trees
3. Optimize Weights: Uses PSO to find optimal tree weights that maximize:
   • Classification accuracy (F1-score)
   • Ensemble diversity (1 - weighted kappa)
   • Feature entropy (Shannon entropy of root features)

The multi-objective optimization allows WRFO to select a diverse, accurate subset of trees while maintaining interpretability through feature diversity.

Algorithm Parameters

Citation

If you use WRFO in your research, please cite:

@article{wrfo2026,
  title={Towards Better Random Forests with Tree Weighting, Accuracy and Diversity-Preserving Pruning},
  author={Nour Elislem Karabadji, Ali Assi, Abdelaziz Amara Korba, Ahmed Abdulaziz Al Nuaim, Hassina Seridi, Mohamed Elati, Wajdi Dhifli},
  journal={Expert Systems with Applications},
  year={2026},
  note={Under revision}
}

License

MIT License - see LICENSE file for details

Acknowledgments

• Built on top of scikit-learn's RandomForestClassifier
• PSO implementation from pyswarm package
• Developed for research in ensemble learning optimization