mpsboost 0.5.0

Lightweight gradient boosting for Apple Silicon with an MPS/Metal backend

MPSBoost

MPSBoost is an early-stage gradient boosting project for Apple Silicon. Its current accelerated backend uses custom Metal compute kernels for squared-error gradients and two-stage histogram construction while keeping one deterministic tree-building implementation shared with the CPU oracle.

Documentation is available at billzi2016.github.io/MPSBoost.

Development status: 0.5.0 is the current 0.x hardening milestone. It builds on the v2 arboretum and S18 validation releases with sklearn-style classifiers, native CPU multiclass softmax, random forests, ExtraTrees, decision trees, CatBoost-like numeric estimators, advanced regression objectives, feature explanations, CPU-suitable anomaly/ranking estimators, explicit environment guidance, documented backend-selection boundaries, optional SHAP/portable-backend diagnostics, the S18 real-world acceptance report, and user-facing fallback behavior for external backend policies.

Project origin

MPSBoost was started by a Purdue CS PhD student working across algorithms, systems, AI, compilers, and formal verification. The immediate motivation is practical: Apple Silicon has a strong GPU stack, but common tree-based machine learning workloads still lack a simple, fast, low-permission MPS-accelerated path.

The project is built with an SDD workflow. Work is split into clear stages: first validate real MPS/Metal kernels and runtime behavior, then lock the specs and product requirements, then settle the technical stack, and finally execute the task list against those specs. Specs are treated as the project contract, not as after-the-fact notes.

For AI agents and automation, mps_boost_skill.md is the canonical usage entry point. It describes the complete target usage pattern, import style, estimator names, backend policy, sklearn model selection, model persistence, diagnostics, and implementation constraints.

Installation

python -m pip install mpsboost

Accelerated releases provide prebuilt Apple Silicon wheels; normal users will not need a heavyweight framework, package manager, CMake, or a local Metal shader compiler.

Estimator-style API

import mpsboost as mb

model = mb.GradientBoostingRegressor(
    n_estimators=200,
    learning_rate=0.05,
    max_depth=6,
    device="mps",
)

model.fit(X_train, y_train)
prediction = model.predict(X_test)
importance = model.feature_importances_
permutation = model.permutation_importance(X_test, y_test)
model.save_model("model.mb")

restored = mb.GradientBoostingRegressor(device="mps")
restored.load_model("model.mb")

MPSBoostRegressor remains available as a backwards-compatible project-branded alias for the same implementation.

fit(..., sample_weight=...) and score(..., sample_weight=...) are supported by the shared CPU/MPS training path. Weights are applied to native gradients, Hessians, split gains, leaf values, and default estimator scores rather than handled as a Python-only post-processing step.

sklearn model selection

MPSBoost estimators are designed to follow the sklearn estimator protocol, so users should be able to use the standard sklearn model-selection stack instead of learning a project-specific search API.

from sklearn.model_selection import GridSearchCV
import mpsboost as mb

search = GridSearchCV(
    mb.GradientBoostingRegressor(device="auto"),
    param_grid={
        "max_depth": [3, 6, 9],
        "learning_rate": [0.03, 0.1],
        "n_estimators": [100, 300],
    },
    cv=3,
    n_jobs=2,
)

search.fit(X_train, y_train)
best_model = search.best_estimator_

The same direction applies to RandomizedSearchCV, cross_val_score, and classifier estimators. Current regressors and classifiers expose get_params(), set_params(), fit(), predict(), predict_proba() where applicable, and default score() methods needed by standard sklearn search utilities. Real-world Iris acceptance includes a standard GridSearchCV multiclass run.

Multiprocessing is supported through sklearn/joblib at the outer search level. CPU jobs can run in multiple processes. MPS jobs should be scheduled more conservatively: several Python workers competing for one Apple GPU can be slower than one well-sized GPU worker because command queues, unified memory bandwidth, and synchronization overhead become the bottleneck. The intended policy is to let device="auto" choose CPU for small search jobs and reserve MPS for workloads where the measured tree hot path is large enough.

Tree estimator names

The primary public API uses concise sklearn-style estimator names, so users can usually switch libraries by changing the import and keeping familiar model names.

Model family	Primary names	Status
Histogram gradient boosting	GradientBoostingRegressor	Available
Histogram gradient boosting classification	GradientBoostingClassifier	Available for binary labels and multiclass one-vs-rest
Random forest	RandomForestRegressor, RandomForestClassifier	Available
Extra trees	ExtraTreesRegressor, ExtraTreesClassifier	Available
Single decision tree	DecisionTreeRegressor, DecisionTreeClassifier	Available
CatBoost-like ordered boosting	CatBoostRegressor, CatBoostClassifier	Available for numeric features
Isolation forest	IsolationForest, MPSIsolationForest	Available; CPU is selected for this branch-heavy workload because it is expected to be faster than Apple GPU
Ranking trees	LearningToRankRegressor	Available; CPU is selected for this latency-sensitive workflow because it is expected to be faster than Apple GPU

Objective variants such as quantile, Poisson, Tweedie, logistic, and ranking losses should be selected through estimator parameters when they share the same tree engine. Separate class names are added only when the model family has different fit/predict semantics.

The same status information is available from Python:

import mpsboost as mb

print(mb.available_estimators())
print(mb.planned_estimators())
mb.require_estimator_supported("CatBoostRegressor")

The 0.5.0 milestone delivers the v2 arboretum foundation plus explicit optional SHAP and portable-backend diagnostics: one shared tree-family registry, unified semantics for boosting/bagging/random-split models, honest backend policy, and no placeholder estimator classes.

Random forest row sampling, feature subsampling, ExtraTrees random thresholds, and CatBoost-like ordered permutations share one deterministic randomization contract. Categorical features can be marked with categorical_features=[...]; CatBoost-like estimators also accept cat_features=[...] as an alias. Categories are ordered by training-target statistics and then passed through the same native histogram split engine. Unknown prediction categories are encoded as missing values and use the native missing-value default direction. Categorical model persistence intentionally fails until the public model format stores category mappings.

Validation metric history and early stopping also share one estimator-independent monitoring contract, so future classifiers and tree ensembles do not duplicate callback semantics.

LightGBM-like controlled leaf-wise growth is available through growth_strategy="leaf_wise". The implementation uses the same native tree engine as level-wise boosting and adds explicit controls for max_leaves, max_active_leaves, and min_gain_to_split. Benchmark scripts compare level-wise and leaf-wise end to end; users should prefer the strategy that wins on their workload rather than assuming one growth policy is always faster.

CPU and MPS backends

MPSBoost treats CPU as a first-class backend, not as a temporary fallback. The CPU oracle/backend is implemented inside this project and shares the same quantization, objective, sampling, monitoring, split-gain, and model-format contracts as the MPS backend. MPSBoost does not call XGBoost, LightGBM, CatBoost, or scikit-learn as hidden training engines.

The intended long-term policy is:

• device="cpu" forces the in-project CPU backend.
• device="mps" forces the in-project MPS/Metal backend.
• device="auto" chooses CPU for small or synchronization-heavy workloads and MPS for workloads where the measured tree hot path can dominate transfer and launch overhead.

MPS is an acceleration backend, not a requirement. If CPU is faster or more stable for a given workload, the project should say so and use CPU under auto.

Prediction path

All delivered tree families share one prediction contract. Gradient boosting, single trees, random forests, ExtraTrees, and CatBoost-like numeric estimators all reuse the native flat-tree model format plus a shared Python aggregation helper for feature-count validation, feature-subset slicing, and forest averaging.

Current MPS acceleration targets training hot paths. Prediction uses the shared native tree traversal path and may run faster on CPU for small batches. A future MPS batch traversal kernel can replace the traversal backend without changing estimator APIs or model files.

Backend diagnostics

The native backend exposes non-sensitive device and cache diagnostics:

import mpsboost as mb

print(mb.__version__)
print(mb.is_available())
print(mb.system_info())
print(mb.mps_setup_instructions())
print(mb.cache_info())

If device="mps" is requested on a machine or session where Apple GPU acceleration is unavailable, MPSBoost reports copy-paste setup commands:

xcode-select --install
xcodebuild -downloadComponent MetalToolchain
python -m pip install --upgrade --force-reinstall mpsboost
python -c "import mpsboost as mb; print(mb.system_info())"

CPU-only scripts, managed CI, and multiprocessing model-selection workers can skip import-time GPU environment checks without disabling CPU training:

MPSBOOST_SKIP_ENV_CHECK=1 python your_script.py

cache_info() only reports paths and existence; it does not create directories. create_cache() explicitly creates the L2 cache directories, and clear_cache() safely removes the MPSBoost cache root after rejecting dangerous targets such as the filesystem root, the user home directory, or symlinks. Cache deletion never changes model results.

Optional research and portable-backend dependencies are not installed by default:

python -m pip install 'mpsboost[shap]'
python -m pip install 'mpsboost[xgboost]'
python -m pip install 'mpsboost[sklearn]'
python -m pip install 'mpsboost[cuda]'

mpsboost[shap] is reserved for the official SHAP integration path. Current approximate_shap_values(...) output is a controlled approximation and must not be cited as official SHAP TreeExplainer output. The public helper export_native_trees_for_shap(...) exposes native tree structure for adapter validation without training data, telemetry, credentials, or device identifiers.

Portable backend policy is explicit and observable. Native CPU/MPS remains the default MPSBoost implementation and correctness oracle. Future sklearn/XGBoost/CUDA adapters must be selected through portable policy and report their actual backend in summaries; they are not hidden behind the native MPSBoost backend name.

Project principles

• Familiar XGBoost/scikit-learn-style Python entry points.
• device="mps" as the stable user-facing Apple GPU backend name.
• Custom Metal kernels for tree-specific irregular computation.
• Prebuilt wheels and no heavyweight Python runtime dependency.
• Observable backend policy: invalid inputs fail clearly, while CPU-suitable MPS requests warn and record the backend decision in the training summary.
• End-to-end benchmarks, including preprocessing and synchronization.

Status

The public API currently includes GradientBoostingRegressor, GradientBoostingClassifier, their backwards-compatible project-branded aliases, the estimator capability registry, deterministic randomization and monitoring helpers, cache diagnostics and management helpers, is_available, system_info, and __version__. Training supports dense finite float32/float64-compatible data, ordered categorical feature encoding, feature-level monotonic constraints, path-level interaction constraints, L1/L2/gamma regularization, leaf-value clipping, squared error regression, binary-logistic classification, multiclass one-vs-rest classification, quantile, Poisson, Tweedie, isolation-forest anomaly scoring, pointwise ranking, deterministic quantization, depth-limited histogram trees, sklearn-compatible score(), model save/load for numeric non-categorical binary/native models, gain/split/permutation feature importance, approximate SHAP-like explanations, random forest n_jobs, explicit device="mps", explicit device="cpu", and initial device="auto" selection.

The checked-in S6 benchmark records both regressions and wins. On the M2 Ultra validation machine, small end-to-end training remains slower on MPS, while the gbdt-large-wide scenario reached a 1.629x median speedup with maximum prediction difference around 5.4e-6 versus the CPU oracle.

Sparse matrices, categorical model persistence, public GPU prediction, and full third-party API compatibility are outside this milestone. Small datasets are expected to be slower on the GPU because fixed device setup and synchronization costs dominate; the checked-in benchmark report preserves this regression region alongside larger wins.

Release audits

The 0.5.0 release gate includes:

• CPU, packaging, integration, and real Metal GPU tests on Python 3.10 and 3.13.
• Wheel content checks excluding specs, tests, caches, and build artifacts.
• Dynamic-link checks for the native extension.
• Apache-2.0 project licensing and runtime dependency review.
• Fresh PyPI installation and real MPS smoke validation.

Independence notice

MPSBoost is an independent open-source project. It is not affiliated with, endorsed by, or sponsored by Apple Inc., the XGBoost project, the LightGBM project, the CatBoost project, or the scikit-learn project. The MPS/Metal backend is an independent implementation based on public papers, public API documentation, and original engineering work; it is not derived from those libraries. Apple, Metal, and Metal Performance Shaders may be trademarks of Apple Inc.

License

Apache-2.0