samlb 0.6.0

A unified benchmark framework for evaluating AutoML systems on data streams with fast C++ base algorithms and rigorous prequential evaluation.

Why SAMLB?

Streaming AutoML methods are hard to compare fairly. Different papers use different datasets, evaluation protocols, and algorithm pools. SAMLB solves this by providing:

• Pure C++ core — base learners, preprocessing, feature selection, metrics and drift detection are all native, with no Python ML dependency (Naive Bayes, Hoeffding Trees, KNN, Perceptron, Logistic Regression, and more)
• Framework-agnostic benchmarking -- plug in any streaming AutoML method with just 3 methods
• Standardized prequential evaluation (test-then-train) with windowed metric snapshots for learning curves
• 30 curated datasets (15 classification + 15 regression) spanning real-world and synthetic drift scenarios
• Parallel execution for large-scale experiments across multiple seeds

Installation

From PyPI

pip install samlb

From source

git clone https://github.com/TechyNilesh/samlb.git
cd samlb
pip install -e ".[dev]"

Optional backends

pip install "samlb[vw]"       # Vowpal Wabbit, for the ChaCha regressor
pip install "samlb[river]"    # River algorithms, via the River adapters
pip install "samlb[capymoa]"  # CapyMOA/MOA algorithms (needs a JVM)

Requirements: Python >= 3.9, a C++ compiler (for the native extension), CMake

Quick Start

Python API

from samlb.benchmark import BenchmarkSuite
from samlb.framework.classification.asml import AutoStreamClassifier
from samlb.framework.classification.eaml import EvolutionaryBaggingClassifier
from samlb.framework.random_search import RandomSearch
from samlb.framework.classification.shared_config import (
    SHARED_PREPROCESSORS, SHARED_CLASSIFIER_INSTANCES,
)

suite = BenchmarkSuite(
    models={
        "ASML":         AutoStreamClassifier(seed=42),
        "EvoAutoML":    EvolutionaryBaggingClassifier(seed=42),
        # RandomSearch baseline over the same shared learner pool
        "RandomSearch": RandomSearch(
            scalers=SHARED_PREPROCESSORS,
            models=SHARED_CLASSIFIER_INSTANCES,
            seed=42,
        ),
    },
    datasets=["electricity", "covertype"],
    task="classification",
    n_runs=10,
    window_size=1000,
)
suite.run()
suite.print_table()
suite.to_csv("results/classification.csv")

RandomSearch is task-agnostic: for regression, pass the regression pool (EAML_REG_PARAM_GRID["Scaler"] / ["Regressor"] from samlb.framework.regression.eaml.config) and set clip=True.

Dataset Streaming

from samlb.datasets import stream, list_datasets

# See all available datasets
print(list_datasets("classification"))
print(list_datasets("regression"))

# Stream instance by instance
for x, y in stream("electricity", task="classification"):
    pred = model.predict_one(x)
    model.learn_one(x, y)

CLI

# Full classification benchmark (5 frameworks x 15 datasets x 10 runs)
python examples/run_benchmark.py

# Custom subset
python examples/run_benchmark.py --n_runs 5 --max_samples 50000 --datasets electricity covertype

# Parallel execution across CPU cores
python examples/run_benchmark.py --n_runs 100 --parallel --cpu_utilization 0.8

# Regression benchmark (4 frameworks x 15 datasets x 10 runs)
python examples/run_regression.py
python examples/run_regression.py --n_runs 5 --datasets bike california_housing

Included Frameworks

Classification

Regression

Baseline (classification & regression)

Model Pool Selection: Normal vs. Ensemble Baselines

Every search framework's candidate pool is swappable between two presets — "normal" (plain single models: Naive Bayes, Perceptron, Hoeffding Tree, ...) and "ensemble" (drift-adaptive ensembles: ARF, SRP, Leveraging Bagging, Hoeffding Adaptive Tree) — via get_classification_config / get_regression_config. This is useful for asking a different question than the usual "does search beat RandomSearch": does search still help once every candidate is already a strong, drift-adaptive baseline on its own?

from samlb.framework import get_classification_config
from samlb.framework.classification.asml      import AutoStreamClassifier
from samlb.framework.classification.autoclass import AutoClass
from samlb.framework.classification.eaml      import EvolutionaryBaggingClassifier
from samlb.framework.classification.oaml      import OAMLClassifier

cfg = get_classification_config(pool="ensemble")   # or pool="normal" (default)

model = AutoStreamClassifier(config_dict=cfg.asml_config_dict(), seed=42)
model = AutoClass(config_dict=cfg.autoclass_config_dict(), seed=42)
model = EvolutionaryBaggingClassifier(param_grid=cfg.eaml_param_grid(), seed=42)
model = OAMLClassifier(scalers=cfg.scalers, classifiers=cfg.classifier_instances, seed=42)

Regression works the same way, over ARFRegressor/SRPRegressor instead:

from samlb.framework import get_regression_config
from samlb.framework.regression.asml import AutoStreamRegressor
from samlb.framework.regression.eaml import EvolutionaryBaggingRegressor

cfg = get_regression_config(pool="ensemble")
model = AutoStreamRegressor(config_dict=cfg.asml_config_dict(), seed=42)
model = EvolutionaryBaggingRegressor(param_grid=cfg.eaml_param_grid(), seed=42)

Both presets live in samlb.framework.classification.shared_config (ClassificationConfig) and samlb.framework.regression.shared_config (RegressionConfig) — pass a custom instance of either dataclass to mix and match your own model pool instead of the two built-in presets.

External Algorithms (River & CapyMOA)

Benchmarks often need to place SAMLB's frameworks next to algorithms from River or CapyMOA. Two adapters make any learner from either library usable as a SAMLB model — same predict_one / learn_one / reset contract, so it drops straight into BenchmarkSuite and is scored by the same prequential evaluator.

Both libraries are optional. Nothing is imported until an adapter is constructed, and is_available() lets a suite skip a backend that is not installed rather than fail.

from river import forest, preprocessing
from samlb.benchmark import BenchmarkSuite
from samlb.framework.adapters import CapyMOAClassifier, RiverClassifier
from samlb.framework.base import ARFClassifier

models = {"SAMLB-ARF": ARFClassifier(n_models=10, seed=42)}

if RiverClassifier.is_available():
    models["River-ARF"] = RiverClassifier(
        preprocessing.StandardScaler() | forest.ARFClassifier(n_models=10, seed=42),
        name="River-ARF",
    )

if CapyMOAClassifier.is_available():
    models["MOA-ARF"] = CapyMOAClassifier(
        "AdaptiveRandomForestClassifier", ensemble_size=10, seed=42, name="MOA-ARF",
    )

BenchmarkSuite(models=models, datasets=["electricity"],
               task="classification", n_runs=10).run()

examples/run_external_baselines.py runs exactly this comparison from the command line, for either task:

python3 examples/run_external_baselines.py --task classification --n_runs 10
python3 examples/run_external_baselines.py --task regression --datasets abalone

River adapters

RiverClassifier / RiverRegressor take a River estimator or a pipeline ending in one. The object you pass is a prototype: it is cloned before every run and never trained in place, so one adapter can be reused across seeds and datasets. Pass a zero-argument callable instead when an estimator cannot be cloned.

from samlb.framework.adapters import RiverRegressor

RiverRegressor(preprocessing.StandardScaler() | linear_model.LinearRegression())
RiverRegressor(lambda: forest.ARFRegressor(seed=1), name="River-ARF")

River classifiers return None until they have seen a label; the evaluator counts those instances but does not score them, exactly as it does for OAML's warm-up.

CapyMOA adapters

CapyMOAClassifier / CapyMOARegressor take a CapyMOA class, or its name in capymoa.classifier / capymoa.regressor, plus any learner keyword arguments. A class rather than an instance, because a CapyMOA learner is bound to a MOA Schema at construction and the schema is not known until the stream starts. The adapter derives it from the first instance, builds the learner then, and converts each {feature: value} dict to the dense array MOA expects.

from samlb.framework.adapters import CapyMOAClassifier, CapyMOARegressor

CapyMOAClassifier("HoeffdingTree", grace_period=50, seed=42)
CapyMOAClassifier("AdaptiveRandomForestClassifier", ensemble_size=10)
CapyMOARegressor("AdaptiveRandomForestRegressor", ensemble_size=10)

MOA works in class indices, so the adapter keeps the label mapping and hands back the original SAMLB labels. Labels are discovered as they arrive, against max_classes reserved nominal slots (100 by default; only MOA's per-class memory scales with it). Pass classes=[...] when the label set is known up front — the schema is then exact and an unexpected label raises instead of being silently absorbed.

C++ Base Algorithms

Every per-instance component is implemented in C++ and exposed through thin Python wrappers:

Classification: Naive Bayes, Perceptron, Logistic Regression, Passive Aggressive, Softmax Regression, KNN, Hoeffding Tree, EFDT, SGT

Ensembles: ARF (Gomes et al. 2017, classification & regression), SRP (Gomes et al. 2019, classification & regression), Leveraging Bagging (Bifet et al. 2010), Hoeffding Adaptive Tree (Bifet & Gavaldà 2009, whole-tree simplification — see HoeffdingAdaptiveTreeClassifier docstring)

Regression: Linear Regression, Bayesian Linear Regression, Passive Aggressive, Hoeffding Tree, KNN

Preprocessing: MinMaxScaler, StandardScaler, MaxAbsScaler, VarianceThreshold, SelectKBest (Pearson)

Metrics: Accuracy, MacroF1, MacroPrecision, MacroRecall, MAE, RMSE, R²

Drift detection: ADWIN, EDDM

Pipelines are fused: scaler | selector | model is executed as a single C++ object, so an instance crosses the Python/C++ boundary once per learn_one / predict_one rather than once per stage.

Evaluation Methodology

SAMLB uses prequential evaluation (test-then-train):

1. For each instance in the stream:
   • Predict -- get the model's prediction before seeing the label
   • Evaluate -- score the prediction against the true label
   • Learn -- update the model with the labelled instance
2. Metrics are captured at configurable window intervals for learning curve analysis
3. Runtime is sampled per-instance for performance profiling

Classification metrics: Accuracy, Macro-F1, Macro-Precision, Macro-Recall

Regression metrics: MAE, RMSE, R^2

Datasets

Classification (15 datasets -- 2.5M+ total instances)

Regression (15 datasets -- 1M+ total instances)

Output Formats

results/
  classification/
    summary.json                  # Flat JSON: one row per (framework x dataset x run)
    <dataset>/<framework>/
      run_00.json                 # Raw per-run JSON with full learning curves
      ...
      run_09.json
      aggregate.json              # Aggregated mean +/- std across 10 runs
  regression/
    summary.json
    <dataset>/<framework>/
      run_00.json
      ...
      run_09.json
      aggregate.json

The released repository includes the raw JSON results used for the paper under results/classification/ and results/regression/.

Project Structure

.
├── pyproject.toml             # Package metadata & build config
├── CMakeLists.txt             # C++ build configuration
├── LICENSE                    # MIT License
├── README.md                  # This file
├── CONTRIBUTING.md            # Contributor guide
├── wiki/                      # Wiki page sources (published to the GitHub wiki)
├── scripts/                   # publish_wiki.sh
├── _cpp/                      # C++ source (9 classifiers, 5 regressors)
│   ├── classification/
│   ├── regression/
│   ├── core/                  # Shared headers
│   └── bindings/              # PyBind11 module
├── samlb/                     # Python package
│   ├── __init__.py            # Version: 0.3.0
│   ├── algorithms/            # C++ algorithm Python bindings
│   ├── benchmark/             # BenchmarkSuite orchestrator
│   ├── evaluation/            # PrequentialEvaluator, metrics, results
│   ├── datasets/              # 30 datasets (15 clf + 15 reg NPZ files)
│   └── framework/             # AutoML framework implementations
│       ├── base/              # BaseStreamFramework + C++ wrappers
│       ├── adapters/          # River & CapyMOA adapters (optional backends)
│       ├── random_search.py   # RandomSearch baseline (task-agnostic)
│       ├── classification/    # ASML, AutoClass, EvoAutoML, OAML
│       └── regression/        # ASML, ChaCha, EvoAutoML
├── results/                   # Raw paper results as JSON files
│   ├── classification/        # Classification run_*.json + aggregate.json
│   └── regression/            # Regression run_*.json + aggregate.json
├── tests/                     # Test suite
└── examples/                  # Benchmark runner scripts
    ├── run_benchmark.py       # Classification benchmark CLI
    ├── run_regression.py      # Regression benchmark CLI
    └── run_external_baselines.py  # River / CapyMOA comparison CLI

Documentation

Full usage documentation lives in the SAMLB wiki:

The pages are version-controlled in wiki/ — edit them there and open a PR; ./scripts/publish_wiki.sh pushes them to the wiki.

Contributing

Contributions are welcome — a new streaming AutoML framework, a dataset, an adapter for another library, a bug fix, or a documentation correction.

git clone https://github.com/TechyNilesh/samlb.git
cd samlb
pip install -e ".[dev]"
pytest tests/
ruff check samlb/

Adding a framework means implementing three methods:

from samlb.framework.base import BaseStreamFramework

class MyStreamingAutoML(BaseStreamFramework):
    def predict_one(self, x): ...   # predict BEFORE learning
    def learn_one(self, x, y): ...  # your AutoML logic lives here
    def reset(self): ...            # back to untrained; called before every run

CONTRIBUTING.md has the full walkthrough — development setup, rebuilding the C++ extension, the step-by-step guide to adding a framework, dataset, adapter or C++ learner, and the PR checklist.

Citation

If you use SAMLB in your research, please cite:

@inproceedings{verma2026samlb,
  title     = {SAMLB: A Streaming AutoML Benchmark},
  author    = {Verma, Nilesh and Bifet, Albert and Pfahringer, Bernhard and Bahri, Maroua},
  booktitle = {Proceedings of the International Conference on Automated Machine Learning (AutoML)},
  year      = {2026},
  url       = {https://github.com/TechyNilesh/samlb}
}

License

MIT License. See LICENSE for details.