fastforest

Fast approximate-forest regression and multiclass classification in Rust, with Python bindings. It can quickly and accurately fit datasets with arbitrarily large row counts (millions of rows or more), and scales down to tiny datasets too.

Across nineteen numeric and mixed-data benchmarks spanning 1,030 to 20,216,100 rows and covering regression, binary classification, and multiclass classification, FastForest is always either the fastest to train and predict, or the most accurate. For more results, see the benchmarks section.

Regression

_{Bold is best for that dataset and metric. AutoForest includes automatic sample sizing; autogrow additionally sizes the forest. All rows were measured on an Apple M5 Max; fit includes preprocessing.}

Classification

F1 acc is macro-averaged F1, giving every class equal weight. Covertype is passed with its supplied binary features; FastForest bundles exclusive indicators automatically.

Install

pip install fastforest

This installs the Python library and the native fastforest-fit, fastforest-predict, fastforest-convert, and fastforest-compile executables.

Training and Validation

rng = np.random.default_rng(42)
X = rng.random((1_000, 6))
y = 4*X[:, 0] - 2*X[:, 1] + X[:, 5]

ff = FastForest(seed=42, oob=True).fit(X, y)
preds = ff.predict(X[:5])
preds

array([3.0069957, 2.3236141, 1.0881499, 2.9502003, 3.3705614],
      dtype=float32)

labels = np.where(X[:, 0]+X[:, 1] > 1, "high", "low")
ffc = FastForestClassifier(seed=42, oob=True).fit(X, labels)
probs,classes = ffc.predict_proba(X[:5]), ffc.predict(X[:5])
classes

array(['high', 'high', 'high', 'high', 'low'], dtype='<U4')

X may contain numeric values, numeric strings, ordinary strings, and configured missing values. Regression y is converted to contiguous float32 and must be finite. Classification labels may be numeric or strings; classes_ records their probability-column order. Missing labels and single-class targets are rejected.

Automatic sizing

AutoForest and AutoForestClassifier size the samples while retaining the ordinary estimator API; autogrow=True also sizes the forest:

from fastforest.auto import AutoForest,AutoForestClassifier

model = AutoForest(seed=42).fit(X, y)
classifier = AutoForestClassifier(seed=42).fit(X, labels)
grown = AutoForest(autogrow=True, seed=42).fit(X, y)

For sufficiently large data, one parallel eight-tree screen tries only larger bootstrap_max and max_node_samples values. Each extra level requires another 1% reduction in OOB loss, independently on each axis. Ordinary sizing tries bootstrap limits of 80k, 120k, 160k, and 200k and node samples of 640, 960, and 1280; autogrow widens these to 80k, 160k, 240k, and 320k and 640, 1280, and 1920. When the row cap removes wider bootstrap choices, their vacant comparison slots are filled from the ordinary grid. The screen is skipped unless rows exceed 2 * bootstrap_max * max(1, classes-1), so the default thresholds are 80,000 rows for regression and binary classification, 160,000 for three classes, and 480,000 for seven classes.

By default, the final model uses fastforest’s ordinary adaptive 32–64 tree rule and does not enable OOB. With autogrow=True, it instead grows in 32-tree batches. An independent random set of at most 40,000 tracking rows per output is fixed before the first batch; at every checkpoint, each row uses only trees for which it was out-of-bag. Another batch is added while cumulative regression MSE or classification Brier loss improves by at least 1%; the first batch that fails this test is discarded by default. Growth is capped at 512 trees by default; keep_last_batch, min_improvement, tree_batch_size, and max_trees control these choices.

Creating and Using Models

Models can be saved as compact, portable .ffm files containing the forest, fitted preprocessing schema, task, and class labels. A loaded model supports ordinary in-memory prediction as well as bounded file prediction:

from fastforest import load

model.save("model.ffm")
restored = load("model.ffm")
predictions = restored.predict(X)
restored.predict_file("test.csv", "predictions.csv")
restored.save_executable("model-predict")

predict_file processes CSV or Arrow IPC/Feather in bounded batches rather than loading the whole input. save_executable builds a standalone predictor for the current platform, embedding both the model and Rust prediction runtime; building it requires a Rust toolchain, but running it requires neither Python nor a separate model file.

Installing fastforest also provides four commands. Their parsing, preprocessing, fitting, persistence, and prediction run in Rust:

fastforest-fit train.csv --target price --task regression --output model.ffm
fastforest-predict model.ffm test.csv --output predictions.csv
fastforest-convert numeric.csv --output numeric.arrow
fastforest-compile model.ffm --output model-predict
./model-predict test.csv --output predictions.csv

fastforest-fit accepts mixed CSV or numeric Arrow input and supports regression and classification; classification prediction accepts --proba. fastforest-convert streams numeric CSV into standard Arrow IPC for faster repeated ingestion. Run any command with --help for its complete estimator, schema, and batching options.

The native fastforest-predict binary, using a default model trained on an 80% Concrete Strength split, predicts from Arrow end-to-end in 4.5 ms for one row and 4.9 ms for all 206 validation rows. Reproduce it with python tools/cli_bench.py.

Benchmarking

This section contains additional results; all benchmarks, including those at the top of the README, follow the approaches described here. Unless noted otherwise, results use one reproducible 80/20 split, stratified for classification. Fit timing includes model construction, schema inspection, preprocessing, and fitting, but excludes process startup and inter-process transfer. Prediction timing includes input transformation. Every model/dataset combination has a 180-second limit.

Each AutoForest row uses the ordinary adaptive tree count. Its following autogrow row uses the same sample sizer with growth capped at 192 trees. Both include the sizing screen and final fit in fit time, and appear only when training rows exceed the sample-sizer activation threshold.

Regression

For mixed data, the sklearn benchmarks use a custom pipeline based on scikit-learn’s official preprocessing guidance and examples: wholly numeric columns are parsed and median-imputed, categorical columns use one-hot encoding through 20 levels and target encoding above that, and HistGBM uses native categoricals through its 255-level limit. This numeric parsing is needed for sensible handling of raw CSV-like tables; otherwise the pipeline uses the documented sklearn behavior. fastforest requires no custom preprocessing and takes the original datasets directly. A timed-out cell marks a model that exceeded the 180-second per-model limit. For validation, Blue Book uses its final 12,000 rows, Walmart uses a 12-week chronological holdout to match the competition’s future-period forecasting setup, Rossmann uses its final six weeks, and ASHRAE uses December 2016. On those four datasets the FastForest models set order= to the split column. The Target statistics section under Data preparation describes what the declared order changes.

Classification

Reproducing the benchmarks

Install the development dependencies and release build, then reproduce one dataset with:

pip install -e '.[dev]'
cargo build --release --bins
python tools/stage_binaries.py
maturin develop --release
python tools/accuracy.py --dataset california

Available regression datasets are sgemm, california, concrete, diamonds, allstate, diabetes, bluebook, bluebook_raw, walmart, walmart_raw, ashrae, and rossmann. Classification choices are covertype, adult, bank, click, shuttle, airlines, higgs, sf_police, and kddcup99. Run one forest alone with --ff_only, --auto_only, or --rf_only, or reproduce all displayed results with:

for dataset in sgemm california concrete diamonds allstate diabetes; do
  python tools/accuracy.py --dataset "$dataset"
done

for dataset in covertype adult bank click shuttle airlines higgs sf_police kddcup99; do
  python tools/accuracy.py --dataset "$dataset"
done

python tools/accuracy.py --dataset walmart --ff_only

Data preparation

FastForest fits a deterministic schema for every input column:

1. Non-missing values are parsed as float32 when every value can be parsed and are otherwise treated as strings. Numeric columns sort numerically and other columns sort lexically. Numeric columns whose values are all integral retain that metadata so analysis displays them with no decimal places.
2. A constant column is discarded. Every other column becomes one zero-based rank in its sort order; binary columns are therefore ordinary boolean features.
3. The default missing value is the empty value. Override it per column with missing_values, using column names or indexes. Missing is encoded as a separate rank, and each split learns whether it belongs in its left or right child. No imputation or indicator column is added. By default, a column containing no training missing values rejects missing values during prediction; set allow_new_missing=True to route them to the larger child seen in the split’s sampled rows. Entirely missing columns are discarded.

X = np.array([
    ["18", "red",   ""],
    ["42", "blue",  "3.5"],
    ["31", "green", "2.0"],
], dtype=object)

model = FastForest(missing_values={2: ""}).fit(X, [1, 4, 3])

Binary columns with no missing values are checked for mutual exclusivity on at most 10,000 sampled training-pool rows. Compatible indicators are collapsed into one categorical feature when their bundle is active in more than half the sample. The fitted membership and order are saved with the model; importance, explanations, and partial dependence treat the bundle as one feature and column_info_ lists its members.

Date and time columns are detected by default from at most 200 random training-pool rows using a conservative list of common formats. Every sampled non-missing value must match; ambiguous day/month forms remain candidates until a value above 12 resolves them, with month-first used if they remain ambiguous. Detected formats are saved with the model and never inferred again during prediction. Date columns are expanded natively using the same parts as fastai’s add_datepart: year, month, ISO week, day, day-of-week, day-of-year, month/quarter/year boundary flags, hour, minute, second, and Unix elapsed seconds. Constant parts are discarded automatically, while missing or unparsable date values produce ordinary missing date parts.

Set date_columns={} to disable detection, or provide explicit strftime formats to override it:

model = FastForest(date_columns={"saledate":"%m/%d/%Y %H:%M"}).fit(X, y)

Ranking is a compact training representation, not a prediction-time requirement for numeric columns. After fitting, rank cutoffs are converted back to native numeric boundaries, so seen and unseen numeric values are compared directly without a rank lookup. Nonnumeric values are mapped through their fitted lexical ordering, with unseen values receiving their insertion rank. Missing numeric values remain NaN during native prediction and follow the direction stored in each split.

Python accepts pandas data frames, NumPy arrays, and Arrow tables, selects the bounded training pool first, converts only retained rows, and performs the bounded 200-row date-format check. The native CSV path likewise builds Arrow arrays only for retained rows, while Arrow IPC keeps its existing typed buffers. Full-column schema fitting and inference transformation then run in Rust behind the Arrow boundary, including numeric and lexical interpretation, missing values, categories, date expansion, and parallel column processing. The compact ranked training matrix and native-value prediction matrix remain internal implementation details.

Generated ranks and date parts remain internal. Feature importance, explanations, and partial-dependence results aggregate them back to the original column and display its original values. Fitted interpretations are available in model.column_info_.

For reproducible sklearn comparisons on the same raw dataframe, sklearn_preprocessor implements the policy used by the benchmark: wholly numeric columns are parsed and median-imputed, categorical columns are one-hot encoded through 20 levels and target encoded above 20, and explicitly supplied missing markers are converted to nulls.

from sklearn.ensemble import RandomForestRegressor
from sklearn.pipeline import make_pipeline
from fastforest import sklearn_preprocessor

preprocess = sklearn_preprocessor(X_train, missing_values={"age":"?"})
model = make_pipeline(preprocess, RandomForestRegressor(n_jobs=-1))
model.fit(X_train, y_train)

Install the optional dependencies with pip install 'fastforest[sklearn]'.

Target statistics

By default, fitting also builds one frozen target statistic per qualifying categorical column. The statistic for a level with at least min_rows_per_level training rows is its mean target (regression) or positive rate (binary classification). The statistic for every other level, and for values unseen during training, is the training mean. Missing is an ordinary level, with its own mean and its own count.

FastForest keeps a column’s statistic only when the two halves of the training data rank its levels the same way, measured by weighted rank correlation of at least min_stat_agreement. With order= naming the column your rows arrive by, the halves are earlier and later rows; without it they are random. On a chronological split, identity columns such as a store id keep their statistics and calendar columns lose them.

model = FastForest(order="saledate").fit(X, y)

The fitted table, saved with the model, is identical at training, out-of-bag evaluation, and prediction. frequency=True adds each level’s exact training count as another derived feature. natural_sort=True ranks digit runs inside strings numerically. target_statistics=False disables the statistics. Multiclass classification builds no target statistic. Importance, explanations, and partial dependence aggregate the derived features back to the original column, as they do for ranks and date parts.

Algorithm

Each regression tree draws min(floor(bootstrap_fraction * n_rows), bootstrap_max) training rows. Classification treats bootstrap_max as a per-output cap and therefore uses bootstrap_max * max(1, n_classes-1) total rows per tree. replacement=None adaptively samples with replacement below 10,000 regression rows or 40,000 classification rows, and otherwise without it; pass True or False to override this. When bootstrap_fraction=None, it resolves to 0.8 with OOB enabled and 1 otherwise. Fractions above 1 are supported with replacement; without replacement the maximum is 1. Pass bootstrap_max=None to disable the cap. At each node, the default histogram splitter:

1. A node with fewer than min_node_size rows, or whose first max_node_samples sampled targets are equal, becomes a leaf.
2. A random contiguous window containing at most max_node_samples of the node’s shuffled rows is selected.
3. The tree randomly selects the configured fraction of encoded features, with a minimum of one.
4. For each selected feature, the sampled rows are sorted by their encoded rank and every distinct observed boundary is evaluated. Regression minimizes size-weighted sample standard deviation, shrinking each child mean toward its parent by a three-row prior. Classification uses tree-frequency-weighted entropy. Missing values occupy the final contiguous rank range: the ordinary pass leaves them right, and a second ordered pass tries them left only when that range is nonempty. These scores penalize poorly supported small children directly; the only hard requirement is that both children are nonempty.
5. Every regression leaf predicts the mean target of all tree-sampled rows that reached it. A classification leaf stores their class-probability vector. Thus leaf fitting processes each tree’s capped sample once in total; it does not route the whole dataset through every tree.

By default, forest size targets two million sampled rows across its trees: n_trees = clamp(ceil(2_000_000 / sampled_rows_per_tree), 32, 64). Set n_trees to override it. The standard regression cap resolves to 50 trees; Covertype’s seven-class cap resolves to 32. Other defaults are minimum node size 8, all rows capped at 40,000 per output, 90% feature sampling for regression or 60% for classification, at most 320 evaluated rows per node, and a three-row regression split prior. Enabling OOB changes the default sampling fraction to 0.8 so every row can receive held-out predictions. Preprocessing and trees build in parallel over columns and trees respectively. Classification prediction divides rows into roughly four blocks per Rayon worker and calculates how many fitted trees fit in a conservative 512 KiB working-set budget, including nodes and leaf probabilities. It processes those cache-sized tree batches within each row block; small trees retain row locality, while large trees automatically become tree-major. Supplying seed makes the fitted forest deterministic regardless of parallel scheduling.

max_features accepts "sqrt" or a fraction in (0, 1]; its default is 0.9 for regression and 0.6 for classification.

FastForestClassifier.predict_proba averages the leaf probabilities over trees, while predict returns the corresponding original label. With OOB enabled, oob_decision_function_, oob_counts_, and OOB accuracy oob_score_ are available; oob_indices_ maps the bounded results to original training rows. Ordinary fitting remains bounded by the shared pool, per-output row cap, and max_node_samples rows per node.

Split selection and tuning

The histogram splitter is the production default. The original random-cutoff search remains available as a simpler teaching implementation:

model = FastForest(random_splitter=True, seed=42).fit(X, y)
fixed = FastForest(max_features="sqrt", seed=42).fit(X, y)

The histogram search randomly selects max_features, builds sparse target-statistic histograms from the node evaluation window, and checks every observed boundary for those features. The random splitter instead proposes random (feature, value) cutoffs, deduplicates them, and evaluates them on the same kind of node window. Its candidate count is controlled by cutoff_divisor; max_features is ignored when random_splitter=True.

The focused sweep tool takes comma-separated levels for every tree hyperparameter. The first value is the shared baseline and each later value creates one one-axis configuration. It compares an eight-tree batched OOB screen with ordinary resolved-tree fits on the dataset’s canonical validation split, recording OOB, validation, and both training losses in one per-dataset CSV:

python tools/sweep.py --dataset california

Out-of-bag predictions

OOB calculation is opt-in with oob=True. After fitting:

• oob_prediction_ contains each training row’s mean prediction from trees that did not sample that row.
• oob_counts_ contains the number of contributing trees.
• A row with no contributing tree has count zero and prediction NaN.
• Sampling without replacement at bootstrap_fraction=1.0 leaves no OOB rows, so all counts are zero and predictions are NaN.

Both attributes are None when OOB is disabled.

Model analysis

FastForest includes analysis tools with ordinary NumPy results. Data frames are accepted and supply feature names automatically; arrays use x0, x1, and so on. Sampling happens before Arrow conversion: permutation importance and feature relations use at most 5,000 rows, PDP/ICE uses 500, feature dependence uses 5,000, and drop-column importance uses at most 40,000 training and 5,000 validation rows by default. These limits are configurable through each function’s sampling arguments. Plot methods import matplotlib only when called.

The executable examples below use the 1,030-row Concrete Compressive Strength dataset, cached under data/.

Xc,yc = fetch_openml(data_id=44959, return_X_y=True, as_frame=True, data_home="../data")
Xc_train,Xc_valid,yc_train,yc_valid = train_test_split(Xc, yc, test_size=.2, random_state=42)
concrete = FastForest(seed=42, oob=True).fit(Xc_train, yc_train)

Importance

Use validation-set permutation importance by default. It measures the drop in model score after shuffling a feature without retraining:

importance = concrete.feature_importance(Xc_valid, yc_valid)
importance.plot();

Correlated features can substitute for one another and therefore look individually unimportant. Permute them together to measure their joint importance:

importance = model.feature_importance(X_valid, y_valid,
    features={"location": ["latitude", "longitude"]})

model.drop_column_importance(X_train, y_train, X_valid, y_valid) performs the slower complementary analysis: it refits the forest without each feature. It accepts the same features groups. model.split_importance() returns the nearly free, normalized training-time split-gain measure, but permutation or grouped permutation is preferable because split importance is biased by the available cutoffs and correlated predictors.

Individual predictions and uncertainty

explanation = concrete.explain(Xc_valid[:3])
explanation.row(0)

[('age', 365, 9.071619033813477),
 ('fine_aggregate', 670.0, 5.62085485458374),
 ('water', 228.0, -2.9426684379577637),
 ('blast_furnace_slag', 114.0, 2.7738962173461914),
 ('superplasticizer', 0.0, -1.4535503387451172),
 ('coarse_aggregate', 932.0, 0.7147500514984131),
 ('cement', 266.0, -0.1426078975200653),
 ('fly_ash', 0.0, 0.08987802267074585)]

explanation.plot(0);

tree_predictions = concrete.predict_trees(Xc_valid)
prediction_std = concrete.predict_std(Xc_valid)
prediction_std[:5]

array([ 3.8229113, 10.553153 ,  8.164038 ,  6.691792 ,  5.368883 ],
      dtype=float32)

For every row, prediction = bias + contributions.sum(). Contributions telescope through each tree’s decision path and are then averaged across trees. They explain this forest’s computation, not causality; correlated features can redistribute contributions between themselves.

Partial dependence and ICE

age = concrete.partial_dependence(Xc_train, "age")
age.plot();

age.plot(clusters=5);

interaction = concrete.partial_dependence(Xc_train, ["cement", "water"])
interaction.plot();

Partial dependence repeatedly replaces the selected feature values and averages the resulting predictions. ICE retains the individual prediction lines. These plots describe the fitted model rather than a causal intervention, and highly correlated features can produce unrealistic synthetic rows.

Grouped features aggregate as one feature:

enclosure = model.partial_dependence(X_train,
    {"enclosure": ["enclosure_ac", "enclosure_erops", "enclosure_orops"]})

Collinearity and redundancy

relations = feature_relations(Xc_train)
relations.groups(threshold=0.2)

[('cement',),
 ('blast_furnace_slag',),
 ('fly_ash',),
 ('water',),
 ('superplasticizer',),
 ('coarse_aggregate',),
 ('fine_aggregate',),
 ('age',)]

relations.plot_dendrogram();

relations.plot();

feature_dependence complements correlation: it measures how predictable each feature is from the others, in any nonlinear form the forest can capture.

dependence = feature_dependence(Xc_train)
dependence.predictability

array([ 0.92725313,  0.86571008,  0.93969655,  0.89854634,  0.93675119,
        0.9088245 ,  0.92628348, -0.02972996])

dependence.plot();

feature_relations uses tie-aware Spearman correlation and average linkage implemented directly with NumPy. feature_dependence detects nonlinear redundancy by treating each feature in turn as a target, fitting a small forest from the remaining features, and measuring grouped prediction and permutation dependence.

Development

The project is locally installed with maturin until it joins the aai-ws workspace:

cargo build --release --bins
python tools/stage_binaries.py
maturin develop
cargo test
pytest -q

For performance work, build the extension in release mode and run the benchmark:

maturin develop --release
python tools/bench.py

Compare accuracy and timings against sklearn’s random forest and histogram GBM on one fixed California Housing split:

python tools/accuracy.py

README.md is generated from nbs/index.ipynb. The displayed results live in tools/results/. After updating those CSVs, re-execute the notebook and run nbdev-readme.

Use --dataset concrete for the smaller Concrete Compressive Strength regression dataset, or --dataset sgemm for the 241,600-row SGEMM GPU Kernel Performance dataset. Each model/dataset combination runs in an isolated process with a three-minute timeout; process startup and input transfer are excluded from reported timings.

Use --ff_only with --min_node_size, --bootstrap_fraction, --bootstrap_max, --replacement, --max_node_samples, and --cutoff_divisor for focused FastForest experiments. These spellings come directly from the call_parse function parameters.