forelint

Lint your forecast backtests before you trust a score.

forelint checks the things a backtest score hides: leakage between related series, the run-to-run noise floor, whether model A is really better than model B, and which items carry the bias and the error. It works on the long-format tables that statsforecast, mlforecast and most forecasting code already produce, and on any scikit-learn estimator.

pip install forelint

What it checks

Quickstart

import forelint as fl

# df: one row per item and date, the actual in "y", one column per model,
# e.g. the output of statsforecast's cross_validation()
report = fl.audit(df, models=["LightGBM", "ETS"], baseline="SeasonalNaive")
print(report)

# Is LightGBM really better than the seasonal naive, or is it noise?
print(fl.dm_test(df, "LightGBM", "SeasonalNaive"))

# How much does random K-fold flatter the model, compared with holding out
# the future of entities the model never saw?
fl.split_optimism(model, X, y, groups=store_ids, time=dates)

# What difference between two runs is just noise?
nf = fl.noise_floor(lambda seed: train_and_score(seed), seeds=range(10))
nf.is_meaningful(new_mae - old_mae)

On real data: one Walmart store in M5

benchmarks/m5_benchmark.py runs every check on the public M5 data: store CA_1, department FOODS_3, 823 items, the last 365 days, a direct 28-day-ahead gradient-boosting model from scikit-learn, scored on the final 28 days. Measured, not assumed:

• The model added nothing over a 28-day moving average. Forecast value added was −0.1% on MAE, the model was worse than the moving average on 49% of items, and Diebold-Mariano gives p = 0.31. The model confidence set keeps both. An average-MAE table alone would not show this.
• It under-forecasts by 9.3% of actual sales.
• 10% of items hold 73% of the squared error.
• The noise floor is small here: over 10 seeds the MAE sd is 0.0014, so any change under 0.0045 is noise.
• Splitting barely mattered for this data: random K-fold was 1% optimistic against a time split. Leakage depends on the features; forelint measures it rather than assuming it.

The model is deliberately plain (six features, no tuning). The point is the audit, not the model.

Why

In an international machine-learning competition on well-geology prediction, validation ranked five builds against the hidden leaderboard with a correlation of −0.745: the build it liked most scored worst, and identical code scored 6.241, 6.259 and 6.368 on three runs. Both failures were measurable before the deadline with the right checks. forelint is those checks, packaged.

Background: My validation ranked five models backwards.

Conventions

• Error is forecast − actual. Positive bias means over-forecasting.
• Tracking signal is the sum of errors divided by the mean absolute error; the default alarm level is ±4.
• dm_test averages the loss difference across series at each time stamp, then applies Diebold-Mariano with the Harvey-Leybourne-Newbold correction.
• model_confidence_set follows Hansen, Lunde and Nason (2011) with the T_max statistic and a moving-block bootstrap. Its kept set matches arch.bootstrap.MCS in the test suite.

Status

Alpha. The API may change before 1.0. Issues and pull requests are welcome.

Author

David Mashiah. MIT licensed.