temporalcv 1.0.0

Temporal cross-validation with leakage protection for time-series ML

temporalcv

Temporal cross-validation with leakage protection for time-series ML.

---

Why temporalcv?

Time-series ML has a leakage problem. Standard cross-validation doesn't respect temporal order, and even "proper" walk-forward implementations often miss subtle bugs:

• Lag features computed on full series (leaks future information)
• No gap between train and test (target leaks into features)
• Thresholds computed on full series (future information in classification)

temporalcv provides validation gates that catch these bugs before they corrupt your results.

---

Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                         VALIDATION PIPELINE                             │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│   Data + Model                                                          │
│        │                                                                │
│        ▼                                                                │
│   ┌──────────────────────────────────────────────────────────────┐     │
│   │                    VALIDATION GATES                          │     │
│   │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐        │     │
│   │  │  Shuffled    │  │  Temporal    │  │  Suspicious  │        │     │
│   │  │  Target Test │  │  Boundary    │  │  Improvement │        │     │
│   │  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘        │     │
│   │         │                 │                 │                │     │
│   │         └─────────────────┼─────────────────┘                │     │
│   │                           ▼                                  │     │
│   │              ┌───────────────────────┐                       │     │
│   │              │   HALT / WARN / PASS  │                       │     │
│   │              └───────────────────────┘                       │     │
│   └──────────────────────────────────────────────────────────────┘     │
│                           │                                             │
│          HALT ◄───────────┼───────────► PASS                            │
│            │              │               │                             │
│            ▼              ▼               ▼                             │
│      ┌─────────┐    ┌─────────┐    ┌─────────────────────────────┐     │
│      │ STOP &  │    │  WARN   │    │      CONTINUE TO:           │     │
│      │INVESTIGATE│   │  USER   │    │  - Walk-Forward CV          │     │
│      └─────────┘    └─────────┘    │  - Statistical Tests (DM/PT)│     │
│                                    │  - Conformal Prediction      │     │
│                                    │  - Deployment                │     │
│                                    └─────────────────────────────┘     │
└─────────────────────────────────────────────────────────────────────────┘

Gate Priority

Status	Meaning	Action
HALT	Critical failure detected	Stop immediately, investigate
WARN	Suspicious signal	Proceed with caution, verify externally
PASS	Validation passed	Continue to next stage

---

What Makes This Unique

1. Shuffled Target Test — The definitive leakage detector

   • If your model beats a permuted baseline, features encode target position
   • Catches: rolling stats on full series, lookahead bias, centered windows

2. HALT/WARN/PASS Framework — Actionable validation status

   • Not just metrics, but decisions
   • Prioritized: HALT > WARN > PASS

3. Temporal-Aware Conformal Prediction

   • Adaptive conformal for distribution shift (Gibbs & Candès 2021)
   • Approximate coverage for time series (exact guarantees require exchangeability)

4. High-Persistence Metrics — For sticky series (ACF(1) > 0.9)

   • MASE, MC-SS ratio, directional accuracy
   • Standard metrics mislead on near-unit-root data

5. sklearn Integration — Drop-in replacement

   • WalkForwardCV works with cross_val_score, GridSearchCV
   • Proper gap enforcement for h-step forecasting

---

Julia Implementation

The Julia version of this library is available in a separate repository: temporalcv.jl.

It provides native Julia implementations of the same core validation gates and statistical tests.

---

Comparison vs sklearn TimeSeriesSplit

Feature	temporalcv	sklearn	Winner
Gap Enforcement	✅ Native	✅ v1.0+	Both
Window Types	Expanding + Sliding	Expanding only	temporalcv
Leakage Detection	3 validation gates	None	temporalcv
Statistical Tests	DM, PT, HAC	None	temporalcv
Conformal Prediction	Split + Adaptive	External (MAPIE)	temporalcv
Financial CV	Purging + Embargo	None	temporalcv
Split Speed	~0.035 ms	~0.012 ms	sklearn

Key Insight: sklearn's TimeSeriesSplit handles basic temporal splits well. temporalcv adds the validation layer that catches bugs before they corrupt your results.

---

Installation

pip install temporalcv

For development:

pip install temporalcv[dev]

Optional Dependencies

temporalcv has modular dependencies for specific features:

Feature	Install Command	When Needed
Benchmarks	pip install temporalcv[benchmarks]	Running M4/M5 benchmarks
Changepoint	pip install temporalcv[changepoint]	PELT algorithm (requires ruptures)
Model Comparison	pip install temporalcv[compare]	Benchmark runner with DM tests
Development	pip install temporalcv[dev]	Testing, linting, type checking
All Features	pip install temporalcv[all]	Everything above

Core dependencies (always installed):

• numpy >= 1.23.0
• scipy >= 1.9.0
• scikit-learn >= 1.1.0
• pandas >= 1.5.0

Platform Compatibility

Platform	Status	Tested Versions
Linux	✅ Fully supported	Ubuntu 20.04+, Debian 11+
macOS	✅ Fully supported	macOS 11+ (Intel & Apple Silicon)
Windows	✅ Fully supported	Windows 10+, Windows Server 2019+

Python versions: 3.9, 3.10, 3.11, 3.12

CI Matrix: All combinations tested on every PR via GitHub Actions.

---

Quick Example

from temporalcv import run_gates, WalkForwardCV
from temporalcv.gates import gate_shuffled_target, gate_suspicious_improvement

# Validate your model doesn't have leakage
# Step 1: Compute gate results
# Note: n_shuffles>=100 required for statistical power in permutation mode (default)
gate_results = [
    gate_shuffled_target(my_model, X, y, n_shuffles=100),
    gate_suspicious_improvement(model_mae, persistence_mae, threshold=0.20),
]

# Step 2: Aggregate into report
report = run_gates(gate_results)

if report.status == "HALT":
    raise ValueError(f"Leakage detected: {report.summary()}")

# Walk-forward CV with proper gap enforcement
cv = WalkForwardCV(
    window_type="sliding",
    window_size=104,
    horizon=2,  # Minimum required separation for 2-step forecasting
    extra_gap=0,  # Optional: add safety margin (default: 0)
    test_size=1
)

for train_idx, test_idx in cv.split(X, y):
    # Guaranteed: train_idx[-1] + gap < test_idx[0]
    model.fit(X[train_idx], y[train_idx])
    predictions = model.predict(X[test_idx])

---

Features

Validation Gates

• Shuffled target test - Definitive leakage detection
• Synthetic AR(1) bounds - Theoretical validation
• Suspicious improvement detection - >20% = investigate
• Temporal boundary audit - No future in features

Statistical Tests

• Diebold-Mariano test - With HAC variance estimation
• Pesaran-Timmermann test - Direction accuracy (3-class)

Walk-Forward CV

• Sliding and expanding windows
• Gap parameter enforcement
• sklearn-compatible splitter API

High-Persistence Metrics

• MC-SS - Move-Conditional Skill Score
• Move-only MAE - Error when target moved
• Direction Brier - Probabilistic direction accuracy

---

Examples

Real-world case studies demonstrating key features:

Example	Description
01_leakage_detection.py	Shuffled target test catches lookahead bias
02_walk_forward_cv.py	Gap enforcement for h-step forecasting
03_statistical_tests.py	DM test: is improvement significant?
04_high_persistence.py	MASE metrics for sticky series
05_conformal_prediction.py	Adaptive intervals under distribution shift

Interactive Demo:

---

Benchmark Comparison

Feature Matrix

Feature	temporalcv	sklearn	sktime	Darts
Gap enforcement	✅ Built-in	❌ Manual	❌ Manual	❌ Manual
Leakage detection	✅ Gates	❌ None	❌ None	❌ None
Horizon validation	✅ Warnings	❌ None	❌ None	❌ None
Statistical tests (DM)	✅ HAC variance	❌ None	✅ Basic	❌ None
Conformal prediction	✅ Adaptive	❌ None	❌ None	✅ Split
sklearn compatible	✅ Full	✅ Native	✅ Full	❌ Partial

Why Not Just sklearn's TimeSeriesSplit?

from sklearn.model_selection import TimeSeriesSplit

# sklearn: No gap, no horizon validation
cv = TimeSeriesSplit(n_splits=5)  # Target leakage possible for h>1

# temporalcv: Gap enforcement + validation
from temporalcv import WalkForwardCV
cv = WalkForwardCV(n_splits=5, horizon=2, extra_gap=0)  # total_separation = horizon + extra_gap

Benchmark Runner

Compare models across datasets:

from temporalcv.benchmarks import create_synthetic_dataset
from temporalcv.compare import run_benchmark_suite, NaiveAdapter

datasets = [create_synthetic_dataset(seed=i) for i in range(3)]
report = run_benchmark_suite(datasets, [NaiveAdapter()], include_dm_test=True)
print(report.to_markdown())

---

Documentation

Getting Started

• Quickstart Guide - Get started in 5 minutes

Tutorials

• Leakage Detection - Catch data leakage with validation gates
• Walk-Forward CV - Proper temporal cross-validation
• High-Persistence Metrics - Metrics for sticky series
• Uncertainty Quantification - Prediction intervals with coverage guarantees

API Reference

• Validation Gates - HALT/PASS/WARN framework
• Walk-Forward CV - sklearn-compatible temporal CV
• Statistical Tests - DM test, PT test, HAC variance
• High-Persistence Metrics - MC-SS, move-conditional MAE
• Regime Classification - Volatility and direction regimes
• Conformal Prediction - Distribution-free intervals
• Bagging - Time-series-aware bagging
• Event Metrics - Brier score, PR-AUC

Internal

• Planning Documentation
• Ecosystem Gap Analysis

Help & Support

• Troubleshooting Guide - Common issues and solutions
• Testing Strategy - How temporalcv is tested
• Benchmark Methodology - How benchmark results are generated
• GitHub Issues - Report bugs or request features

---

Citation

If you use temporalcv in your research, please cite:

@software{temporalcv2025,
  author       = {Behring, Brandon},
  title        = {temporalcv: Temporal cross-validation with leakage protection},
  year         = {2025},
  publisher    = {GitHub},
  url          = {https://github.com/brandonmbehring-dev/temporalcv},
  version      = {1.0.0}
}

See CITATION.cff for additional citation formats.

---

License

MIT License - see LICENSE

---

Contributing

See CONTRIBUTING.md for guidelines.