preqts

Prequential (test-then-update) streaming evaluation for time-series forecasters.

Real forecasting is streaming: observations arrive over time, covariates interact, some are known into the future, and a model must update its prediction at each step. Public benchmarks evaluate the batch case: at each cutoff they re-forecast an independent window (fev, GIFT-Eval, GluonTS/Darts rolling backtests). None of them evaluate a stateful forecaster that carries hidden state and updates at constant cost, which is exactly the regime of the new recurrent and state-space foundation models (TiRex-2 / xLSTM, FlowState / SSM). When the TiRex-2 authors needed to measure streaming behaviour, they had to hand-roll a bespoke protocol because nothing standard existed.

preqts is that missing harness: it drives any forecaster through a test-then-update loop, scores stateful and batch models identically on one stream, simulates realistic covariate arrival, and measures the per-step compute cost so the streaming-cost argument is verified, not asserted.

Part of the ChronoScope time-series method atlas (extracted as a standalone, PyPI-destined library per CAOS ADR-0061). Core depends on numpy only (Pyodide-safe); model backends are optional extras.

Install

pip install preqts                 # core (numpy only)
pip install "preqts[statsforecast]" # + a batch backend to wrap
pip install "preqts[dev]"           # + pytest, ruff

Quickstart

import numpy as np
from preqts import SeasonalNaive, Stream, run_prequential

rng = np.random.default_rng(0)
y = 50 + 10 * np.sin(np.arange(600) * 2 * np.pi / 12) + rng.normal(0, 2, 600)

stream = Stream(y, seasonality=12, name="demo")
result = run_prequential(SeasonalNaive(12), stream, horizon=1, quantile_levels=[0.1, 0.5, 0.9])

print(result.summary())   # MASE, WQL, SQL, coverage, total_compute_s, final_predict_ms

See examples/quickstart.py for the stateful-vs-batch cost comparison.

Core concepts

Metrics

mase, weighted_quantile_loss (WQL), scaled_quantile_loss (SQL, the probabilistic analogue of MASE used by fev-bench), empirical_coverage, and pinball_loss. MASE and SQL are scaled by the in-sample seasonal-naive MAE fixed from the warmup window, so every cutoff and every model share one denominator.

Wrapping a real batch model

Any callable batch_fn(context, horizon, past_cov, future_cov, quantile_levels) -> (horizon, Q) can be wrapped:

from preqts import ReplayAdapter, run_prequential

def chronos_batch(context, horizon, past_cov, future_cov, levels):
    # call a Chronos-2 / TimesFM / StatsForecast model on `context`, return quantile columns
    ...

result = run_prequential(ReplayAdapter(chronos_batch, name="chronos-2"), stream, horizon=12)

The adapter re-reads the whole history each predict, so result.predict_latency shows the batch cost growing with the stream: the phenomenon a streaming-native model avoids.

Docs

The docs/ wiki explains the method and the design: why prequential (docs/00_why-prequential.md), the protocol and stream model (docs/01_protocol-and-stream.md), the metrics (docs/02_metrics.md), and the conformal module (docs/03_conformal.md).

Status

v0.01.000 (2026-07-03), alpha. API may change while 0.x. 32 tests, ruff-clean.

License

Apache-2.0.