temporalcv 2.3.0

Rigorous cross-validation for time series with leakage detection and gap enforcement.

Documentation • Examples • Leakage Tutorial

Installation

pip install temporalcv

Quick Start

Validate your model for temporal leakage in 4 lines:

from temporalcv import run_gates
from temporalcv.gates import gate_signal_verification

report = run_gates([gate_signal_verification(model, X, y, n_shuffles=100)])
print(report.status)  # HALT, WARN, or PASS

Why temporalcv?

Standard cross-validation shuffles data randomly. For time series, this means training on future data to predict the past — a form of data leakage that inflates metrics.

temporalcv provides:

• Leakage detection via validation gates (shuffled target test, suspicious improvement)
• Gap enforcement for h-step forecasting (no lag-feature contamination)
• High-persistence metrics (MASE, MC-SS) that measure actual skill
• Conformal coverage with caveats: marginal under exchangeability; time-series autocorrelation can invalidate the guarantee — use AdaptiveConformalPredictor (Gibbs & Candès 2021) for distribution-shift handling

Feature Comparison

¹ sklearn TimeSeriesSplit(gap=N) (since v0.24) requires the user to compute N from the forecast horizon themselves; temporalcv derives gap = horizon + extra_gap automatically from the horizon parameter.

Core Features

Walk-Forward CV with Gap

from temporalcv import WalkForwardCV

cv = WalkForwardCV(
    window_type="sliding",
    window_size=104,    # 2 years of weekly data
    horizon=2,          # 2-step ahead forecast
    test_size=1
)

for train_idx, test_idx in cv.split(X, y):
    model.fit(X[train_idx], y[train_idx])
    pred = model.predict(X[test_idx])

The horizon parameter enforces a gap between training and test sets, preventing lagged features from leaking target information.

Validation Gates

from temporalcv.gates import gate_signal_verification, gate_suspicious_improvement

gates = [
    gate_signal_verification(model, X, y, n_shuffles=100),
    gate_suspicious_improvement(model_mae, baseline_mae, threshold=0.20),
]

report = run_gates(gates)
if report.status == "HALT":
    raise ValueError(f"Signal detected — investigate: {report.summary()}")

High-Persistence Metrics

When your series is "sticky" (ACF(1) > 0.9), standard MAE lies — predicting "same as yesterday" looks great but adds no value.

from temporalcv.metrics import mase, mc_skill_score

print(f"MASE: {mase(actual, predicted, y_train):.3f}")
print(f"MC-SS: {mc_skill_score(actual, predicted):.3f}")

Validation Pipeline

flowchart TD
    A[Data + Model] --> B{Validation Gates}
    B -->|HALT| C[Stop & Investigate]
    B -->|WARN| D[Proceed with Caution]
    B -->|PASS| E[Walk-Forward CV]
    E --> F[Statistical Tests]
    F --> G[Deploy]

Common Leakage Patterns

Learn from these failure modes:

See the failure examples gallery for detailed walkthroughs:

• 16: Rolling Stats Leak
• 17: Threshold Leak
• 19: Missing Gap
• 20: KFold Trap — 47.8% fake improvement

Optional Dependencies

pip install temporalcv[benchmarks]   # M4/M5 benchmarks
pip install temporalcv[changepoint]  # PELT algorithm
pip install temporalcv[dev]          # Testing, linting
pip install temporalcv[all]          # Everything

Core: numpy >= 1.21, scipy >= 1.7, scikit-learn >= 1.0, statsmodels >= 0.13, matplotlib >= 3.5 • Optional: pandas >= 1.3 (pip install temporalcv[pandas])

Platforms: Linux, macOS, Windows | Python: 3.10+

Documentation

Validation Evidence

See Testing Strategy for details.

Citation

@software{temporalcv2025,
  author    = {Behring, Brandon},
  title     = {temporalcv: Temporal cross-validation with leakage protection},
  year      = {2025},
  publisher = {GitHub},
  url       = {https://github.com/brandon-behring/temporalcv}
}

Ecosystem

Part of the Rigorous AI Engineering ecosystem:

Contributing

See CONTRIBUTING.md for development setup and guidelines.

License

MIT License — see LICENSE