🚀 MLFastOpt

High-Speed Bayesian Hyperparameter Optimization for ML Ensembles

Installation • Quick Start • Features • Documentation • Contributing

MLFastOpt is a production-ready framework for Bayesian hyperparameter optimization of LightGBM, XGBoost, and Random Forest ensemble models. It combines state-of-the-art Bayesian optimization algorithms with ensemble learning techniques.

✨ Features

📦 Installation

pip install mlfastopt

Prerequisites

• Python: 3.12+
• macOS Users: Install OpenMP for LightGBM/XGBoost support:

  brew install libomp
  

🚀 Quick Start

1. Install the Package

pip install mlfastopt

2. Create Configuration Files

config.json - Main configuration:

{
  "data": {
    "path": "data/train.parquet",
    "label_column": "target",
    "features": ["feature1", "feature2", "feature3"],
    "class_weight": {"0": 1, "1": 5}
  },
  "model": {
    "type": "lightgbm",
    "hyperparameter_path": "config/hyperparameters.yaml",
    "ensemble_size": 10
  },
  "training": {
    "total_trials": 30,
    "sobol_trials": 10,
    "metric": "soft_recall",
    "parallel": true,
    "n_jobs": 4
  },
  "output": {
    "dir": "outputs/runs"
  }
}

config/hyperparameters.yaml - Parameter search space:

parameters:
  - name: learning_rate
    type: range
    bounds: [0.01, 0.3]
    value_type: float
    log_scale: true

  - name: max_depth
    type: range
    bounds: [3, 12]
    value_type: int

  - name: num_leaves
    type: range
    bounds: [20, 150]
    value_type: int

  - name: min_child_samples
    type: range
    bounds: [5, 100]
    value_type: int

3. Run Optimization

MLFastOpt offers two ways to run optimization:

Option A: Command Line (CLI)

# Set OMP_NUM_THREADS=1 to avoid LightGBM/XGBoost deadlocks
OMP_NUM_THREADS=1 mlfastopt-optimize --config config.json

Additional CLI options:

# Validate configuration without running
mlfastopt-optimize --config config.json --validate

# Override trials from command line
mlfastopt-optimize --config config.json --trials 50

# Start web dashboard
mlfastopt-web

# Analysis tools
mlfastopt-analyze

Option B: Python API

from mlfastopt import AEModelTuner

# Initialize with config file
tuner = AEModelTuner(config_path="config.json")

# Run optimization
results = tuner.run_complete_optimization()

# Access results programmatically
print(f"Best parameters: {results['best_parameters']}")
print(f"Output directory: {results['output_dir']}")

4. View Results

Results are saved to outputs/runs/<timestamp>/:

• best_parameters.json — Optimal hyperparameters + metrics (always written)
• qualifying_trials_*.json — All trials meeting the threshold, with per-trial params + metrics
• models/manifest.json — Index of every serialized model file
• models/trial_NNNN_seed_SS.txt — Trained model binaries (LightGBM native format; .pkl for other types)
• optimization_progress.png — Training curves
• feature_importance.png — Feature importance plots
• README.md — Run summary report

📖 How It Works

MLFastOpt uses a two-level nested optimization loop:

┌─────────────────────────────────────────────────────────────────┐
│ OUTER LOOP: Trial Iteration (total_trials = 30)                │
│                                                                 │
│  Trial 1: {learning_rate: 0.05, max_depth: 7, ...}             │
│  ├── Train Model 1 (seed=42)                                   │
│  ├── Train Model 2 (seed=43)                                   │
│  ├── ...                                                        │
│  └── Train Model 10 (seed=51)                                  │
│  └── Ensemble Prediction → Calculate Metrics → Update Optimizer│
│                                                                 │
│  Trial 2: {learning_rate: 0.12, max_depth: 5, ...}             │
│  └── ... (same ensemble training)                               │
│                                                                 │
│  Phase 1: Quasi-random exploration (sobol_trials)              │
│  Phase 2: Bayesian optimization (remaining trials)             │
└─────────────────────────────────────────────────────────────────┘

Key concepts:

• Trial: One hyperparameter configuration tested
• Ensemble: N models trained per trial (different random seeds)
• Soft Voting: Average probabilities across ensemble members
• Hard Voting: Average binary predictions across ensemble members

⚙️ Configuration Reference

Data Section

Model Section

Training Section

Selection Section

Available Metrics

📊 Output Files

After optimization, find results in outputs/runs/<timestamp>/:

outputs/runs/20240205_143022/
├── best_parameters.json        # Best trial's hyperparameters & metrics (always written)
├── qualifying_trials_*.json    # All threshold-qualifying trials (threshold mode)
├── config.json                 # Configuration used for this run
├── optimization_progress.png   # Metric curves across all trials
├── feature_importance.png      # Feature importance chart
├── feature_importance.csv      # Numerical importance data
├── README.md                   # Run summary report
└── models/
    ├── manifest.json           # Index: trial → seed → file path + metrics
    ├── trial_0003_seed_00.txt  # LightGBM native format (.ubj for XGBoost,
    ├── trial_0003_seed_01.txt  #   .pkl for RandomForest)
    └── ...                     # One file per sub-model in each qualifying trial

Loading Saved Models for Inference

import json
import lightgbm as lgb
import numpy as np

# Read the manifest
with open("outputs/runs/<timestamp>/models/manifest.json") as f:
    manifest = json.load(f)

# Load all sub-models for the first qualifying trial
trial = manifest["trials"][0]
models = [lgb.Booster(model_file=sub["file"]) for sub in trial["sub_models"]]

# Ensemble soft-vote prediction
probas = np.mean([m.predict(X_new) for m in models], axis=0)

Why save model files? Metrics reported during optimization reflect ensemble performance (N models averaged together). Deploying the saved ensemble directly guarantees you get the same performance at inference — no re-training required.

📧 Support

For questions, issues, or feature requests, please contact us at contact@genxai.cc.

📄 License

This project is licensed under the MIT License. See the LICENSE file for details.

🏢 About

Developed by GenX AI Lab - Building intelligent AI solutions.