FerrisBoost

FerrisBoost is a Rust-backed gradient-boosted tree library for large tabular data. It uses a column-blocked histogram engine, XGBoost-compatible training semantics for supported objectives, and a Python API with parameter names familiar to XGBoost. Training runs on a single GPU today; the architecture is multi-GPU ready (WIP).

Why FerrisBoost

In large-scale GBDT workflows, GPU acceleration is frequently bottlenecked not by GPU compute kernels, but by host-memory overhead, data ingest, and rigid memory limits. FerrisBoost focuses on reducing ingest, memory, and execution overhead in large tabular workflows:

• Eliminating Host-Memory Bottlenecks & Slow Setup: In XGBoost, GPU training often incurs high host memory and multi-pass materialization when constructing a DMatrix, QuantileDMatrix, or data iterator. FerrisBoost streams and quantizes directly from Parquet or CSV in Rust, making setup 2–8× faster while slashing peak host RAM by 2–4× (0.24–0.48× XGBoost).
• Adaptive VRAM Management: Instead of requiring manual external-memory configurations when datasets exceed physical VRAM, FerrisBoost features an adaptive GPU memory planner that automatically manages residency—seamlessly transitioning between full VRAM residency, hybrid CPU/GPU caching, and streaming execution under memory pressure.
• Optimized for Wide Data (High Feature Counts): Datasets with hundreds or thousands of features (wide tables) severely strain GPU histogram construction and setup time. FerrisBoost's column-blocked histogram engine tiles features into cache-conscious blocks, ensuring bounded GPU-memory planning, strong cache locality, and sustained throughput on high-dimensional datasets.
• Single-GPU Now, Multi-GPU Ready (WIP): Training targets a single selected GPU today (device="cuda" or device="cuda:N"). The column-blocked architecture is engineered to scale across multiple GPUs, with multi-GPU support currently in progress (WIP).
• Deterministic Parity & Flexible Math Modes: FerrisBoost provides deterministic training semantics. Use gpu_math="exact" (default) for byte-identical CPU/GPU models, or gpu_math="fast" for maximum GPU throughput; fast is deterministic but does not guarantee byte-identical CPU/GPU models.
• File-First Rust Data Pipeline: Train directly from Parquet, CSV, or CSV.gz (single or partitioned files), PyArrow tables, or NumPy arrays without constructing an intermediate DMatrix, QuantileDMatrix, or custom iterator. Passing file paths lets Rust inspect schemas and stream-quantize required columns directly.

Performance vs XGBoost

FerrisBoost focuses its performance engineering on GPU acceleration, memory efficiency, and data pipelining. The CPU engine provides a reference implementation with the supported exactness contract, but is not yet micro-optimized.

On the tested workloads, FerrisBoost GPU fast used 0.89–1.34× XGBoost training time, with 0.24–0.48× peak host RAM and 0.39–0.69× combined peak footprint (RAM + active VRAM):

• GPU fast training: 0.89–1.34× XGBoost time (deterministic; maximum throughput)
• GPU exact training: 0.98–1.81× XGBoost time (byte-for-byte CPU-identical)
• GPU setup: 0.12–0.49× XGBoost time — approximately 2–8× faster setup before boosting
• Peak host RAM: 0.24–0.48× XGBoost
• Combined RAM + active VRAM: 0.39–0.69× XGBoost (observed peak footprint during benchmarks; not a hard memory budget guarantee)
• GPU VRAM: Adaptive; full residency when memory is available, hybrid or streaming under tighter budgets
• CPU training: 1.59–2.42× XGBoost time (reference implementation; not yet optimized)

Install

FerrisBoost is available on PyPI:

pip install ferrisboost

One wheel, CPU and NVIDIA GPU support. GPU acceleration is optional; CPU training and inference work without NVIDIA hardware or drivers.

The default package is the portable x86-64 build. Systems that implement the x86-64-v3 ISA level can explicitly install the optimized distribution instead; both distributions provide the same ferrisboost Python API and model format:

# Portable default, including CPU support and optional NVIDIA GPU acceleration:
pip install ferrisboost

# Explicit x86-64-v3 opt-in (install instead of, not alongside, ferrisboost):
pip install ferrisboost-v3

CPU microarchitecture levels are not encoded in stable wheel compatibility tags, so the optimized build uses a separate distribution name rather than a custom platform tag. Do not install both distributions into the same environment.

Install from source

FerrisBoost requires Python 3.9+ and a Rust toolchain when installing from source:

python -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip maturin

# CPU-only build:
maturin develop --release

# CUDA GPU-enabled build (Turing+ / sm_75+; precompiled PTX bundled, CUDA toolkit not required):
maturin develop --release --features "cuda,python,pyo3/extension-module"

Quick start

File-first training & prediction

Pass file paths directly to fb.train() and model.predict(). Python acts as the control plane passing paths and parameters, while Rust handles streaming ingest, column selection, and quantization without materializing full tables in Python:

import ferrisboost as fb

# Train directly from a Parquet file:
model = fb.train(
    {
        "objective": "reg:squarederror",
        "max_depth": 8,
        "nthread": 0,
    },
    "train.parquet",
    label="target",
    num_boost_round=200,
)

# Predict returns a 1-D NumPy float32 array:
prediction = model.predict("test.parquet")
model.save_model("model.json")

Train on GPU:

# Enable GPU training on a selected device with device="cuda" (or "cuda:0", "cuda:1"):
model = fb.train(
    {
        "objective": "reg:squarederror",
        "device": "cuda",
        "max_depth": 8,
    },
    "train.parquet",
    label="target",
    num_boost_round=200,
)

• device: Single GPU today (device="cuda" or device="cuda:N"); multi-GPU scaling is in progress (WIP).
• Adaptive VRAM: Automatically manages GPU memory, dynamically adapting between full residency and streaming execution based on dataset size and available VRAM.
• gpu_math: "exact" (default) guarantees byte-by-byte exact CPU reproducibility; "fast" is deterministic but does not guarantee byte-identical CPU/GPU models.
• nthread: 0 selects available physical CPU parallelism.

Multi-file and partitioned datasets (list of files, directory, or glob pattern):

model = fb.train(
    {"objective": "reg:squarederror", "max_depth": 8},
    ["train_part_0.parquet", "train_part_1.parquet"],  # or "data/partitions/" or "data/*.parquet"
    label="target",
    num_boost_round=200,
)
prediction = model.predict(["test_part_0.parquet", "test_part_1.parquet"])

CSV files with no header use positional column indices:

model = fb.train(params, "train.csv", label=4, header=False)
prediction = model.predict("test.csv", header=False)

Convenience APIs (NumPy & PyArrow)

For in-memory arrays or exploratory workflows where data is already loaded in Python:

import numpy as np
import ferrisboost as fb

rng = np.random.default_rng(42)
X = rng.normal(size=(10_000, 20)).astype(np.float32)
y = (X[:, 0] + 0.5 * X[:, 1] > 0).astype(np.float32)

model = fb.train(
    {
        "objective": "binary:logistic",
        "max_depth": 6,
        "eta": 0.1,
        "nthread": 0,
    },
    X,
    label=y,
    num_boost_round=100,
)

probability = model.predict(X)

PyArrow tables are also accepted:

import pyarrow as pa

table = pa.table({"f0": X[:, 0], "f1": X[:, 1], "f2": X[:, 2]})
model = fb.train(
    {"objective": "binary:logistic", "max_depth": 6, "eta": 0.1},
    table,
    label=y,
    num_boost_round=100,
)

XGBoost Interchange

Because FerrisBoost models are serialized to standard XGBoost-compatible JSON:

• A FerrisBoost GPU-trained model can be loaded by XGBoost on CPU or GPU:

  import xgboost as xgb
  
  booster = xgb.Booster(model_file="fb_gpu_model.json")
  preds = booster.predict(xgb.DMatrix(X))
  

• Similarly, FerrisBoost can load models originally trained by XGBoost (fb.Model.load_model("xgb_model.json")) and predict using FerrisBoost's CPU engine.

In the current release, FerrisBoost model prediction runs single-threaded on the CPU. If inference throughput or latency is a primary concern, models can be loaded directly into XGBoost for multi-threaded CPU or GPU serving.

Known limitations

• CPU prediction: tree scoring is currently single-threaded. Large-batch inference may be faster through XGBoost model interchange.
• Sampling efficiency: subsampling currently preserves full row traversal. On the full-resident column-major GPU path, colsample reduces split enumeration but does not yet skip histogram construction for unselected features.
• GPU inference: post-training prediction currently runs on CPU, including models trained on GPU.
• Multi-GPU: training currently uses one selected GPU per job; multi-GPU execution is not yet implemented.

These are performance and feature limitations, not correctness failures. Future work includes parallel CPU prediction, more selective histogram execution, and multi-GPU support.

Learn more

See the Python API How-To for installation details, Parquet/CSV input formats, validation, early stopping, model persistence, logging, and the complete parameter reference.

License

Apache-2.0.