Forewarm

Token-aware predictive autoscaling for LLM inference.

Forewarm reads vLLM serving metrics from Prometheus, learns the workload's own daily and weekly structure, and publishes a quantile forecast of near-future demand — so replicas can be warmed up before the load arrives instead of after a queue has already formed.

LLM pods cold-start in minutes (weights have to be loaded), which forces reactive autoscalers to keep a guessed, permanently inflated floor of always-on replicas. Forewarm exists to lower that floor safely.

Status

Early development — this release is a placeholder that reserves the name while the first working version is built in the open. There is no usable functionality yet; please do not depend on it.

Planned for the first real release (0.1.0):

• Prometheus ingestion of vLLM metrics (requests, prompt/generation tokens, queue depth)
• Zero-config model fitting on the cluster's own history, with an automatically selected champion benchmarked against a seasonal-naive baseline
• Quantile forecasts exposed as Prometheus metrics and via a gRPC KEDA external scaler
• A backtest CLI reporting forecast accuracy, SLO violation risk, and GPU dollars

Follow the work at https://github.com/Kazantsev-Alexey/forewarm

License

Apache-2.0