force-dataset 1.1.1

FORCE: French Open Research Catalogue of Energy - A High-Performance Python Pipeline

FORCE: French Open Research Catalogue of Energy - A High-Performance Python Pipeline

Version: 1.1.1
License: LGPL-3.0-or-later.
Copyright (c): 2020-2026 EDF (Electricité De France).
Copyright (c): 2023 Sorbonne Université.

Active contributor: Yann Allioux (2020-2026)
Past contributor: Nathan Doumèche (2023-2025)
Scientific advisor: Yannig Goude

Abstract

FORCE provides an extract, load, and transform (ETL) pipeline to aggregate, harmonize, and validate multi-source energy data for the French power system. It addresses the challenge of heterogeneous data by unifying divergent temporal resolutions (e.g., 15-minute vs. 30-minute), spatial granularities, and physical units into a continuous time series matrix.

The package is built for researchers working on time series forecasting, grid economics, and climate-energy modeling. Relying on DuckDB as its analytical backend {cite:p}raasveldt2019duckdb, FORCE consolidates data from four primary sources:

1. RTE (Transmission): A multi-layer architecture tracking data maturity (real-time, consolidated, definitive) to synthesize a single reference record {cite:p}RTE:Eco2Mix.
2. Enedis (Distribution): Granular consumption profiles and decentralized generation metrics {cite:p}Enedis:Bilan.
3. Météo-France (Weather): Synoptic observations transformed via asset-weighted spatial aggregation {cite:p}MeteoFrance:Synop.
4. ODRÉ (Assets): A georeferenced registry of power plants used to generate spatial weights {cite:p}ODRE:Registre.

---

Documentation & Scientific Methodology

Each component of the pipeline is documented with full algorithmic transparency:

Component	Documentation Link	Key Algorithms
Calendar	CALENDAR.md	Skeleton generation, UTC-DST bridge, topological day typing {cite:p}Etalab:Feries,EducationNationale:Calendrier,Augusti:Vacances.
RTE	RTE.md	Multi-version pivoting, conservative upsampling, maturity-aware deduplication.
Enedis	ENEDIS.md	Patch and fill strategy, profile reconstruction.
Météo	METEO.md	Multi-barycenter aggregation, vector decomposition (for circular wind variables).
Radiation	RADIATION.md	PVGIS POA (Plane of Array) & ERA5 GHI (Global Horizontal Irradiance) collection.
Assets	ODRE.md	Two-stage geocoding {cite:p}OpenDataSoft:Georef, spatial centroids.
Imputation	IMPUTATION.md	Pluggable engines, recursive feature relaxation, spatial kriging (interpolation), 7-step causal DAG (Directed Acyclic Graph).
Hierarchies	HIERARCHIES.md	Verified column sum-trees, energy balance, calendar encodings.
Audit	audit/README.md	Per-dataset visualization suite, notebook-engine reproduction.
TAM Extraction	TAM.md	Autoregressive lag/thermal-inertia feature engineering, legacy dataset alignment.

---

Installation & Usage

1. Installation

FORCE requires Python 3.10 or higher (Python 3.12+ recommended for the full TS-ICL imputation engine).

As a package (for end users):

pip install force-dataset

From source (for development or running the pipeline). All dependencies are managed via pyproject.toml.

python -m venv .venv
# activate: PowerShell: .\.venv\Scripts\Activate.ps1 | bash/zsh: source .venv/bin/activate
pip install -e ".[dev]"

Optional extras include dev (tests/lint), docs (Sphinx), dashboard (Jupyter), climate (CDS/CMIP6), and icl (TS-ICL imputation engine, requiring Python >= 3.12).

2. Execution Pipeline

Execute the pipeline steps sequentially to build the dataset:

Phase A: Collection Fetches raw data from APIs. The state-aware engine detects data maturation (e.g., when real-time data is retroactively validated).

energy-collect

Phase B: Transformation Applies cleaning, upsampling, and technical repair. Structural gaps are strictly preserved as NaN.

energy-transform

Phase C: Database Audit Scans DuckDB files for temporal coverage gaps, empty tables, and schema anomalies. For diagnostic visualizations, use energy-viz-transform and energy-viz-assemble.

energy-check

Phase D: Assembly & Imputation Joins sources and applies physics-informed imputation.

energy-assemble
energy-impute  # Uses TS-ICL by default; pass '--engine xgboost' for gradient boosting

---

Scientific Method: Data Harmonization

The pipeline addresses specific challenges in energy data engineering:

1. Temporal Resolution Harmonization (30min to 15min)

The pipeline standardizes all data streams to a 15-minute resolution:

• Energy Flows (Grid Data): We apply a conservative smoothing kernel that mathematically preserves the energy integral (MWh) over the block. Note: Minor numerical deviations may occur when enforcing physical bounds (e.g., clipping negative generation). Formula: $y_{t+15} = \frac{3Y_t + Y_{t+30}}{4}$.
• State Variables (Meteo Data): Instantaneous observations (e.g., temperature) use strict linear upsampling ($limit=1$) to calculate the intermediate point ($t+15$) without modifying the original anchors.

2. Spatio-Temporal Heterogeneity

A simple geographic average of weather stations introduces bias because energy assets are unevenly distributed.

• Spatial Solution: We calculate usage-specific centroids ($G_{load}$, $G_{wind}$, $G_{solar}$) for every region. Station data is aggregated using a modified inverse distance weighting (IDW) algorithm. To ensure numerical stability, this IDW applies a 1 km softening factor and assumes an oblate spheroid Earth ($R=6371$ km).
• Temporal Solution: The pipeline computes time-varying weights ($W_{r,y}$). The influence of a region $r$ on the national signal evolves annually $y$ based on its actual market share. Formula: $I_{national}(t) = \sum_{r} W_{r, y(t)} \cdot I_{r}(t)$

3. Data Maturity & Versioning

Energy data evolves from real-time estimates to validated definitive records. For critical metrics like national load, the pipeline produces multiple time series versions:

• _realtime: For simulating production forecasting.
• _definitive: For training on ground truth.
• _best: A synthesized golden record for general use.

4. Two-Stage Gap Handling Strategy

We separate technical repair from statistical reconstruction to guarantee data integrity:

Stage 1: Technical Repair (ETL Phase)

Targets micro-gaps (< 1h) caused by sensor dropouts.

• Grid Data: Cyclical variables are repaired via a profile-guided bridge ($D-7$) to preserve intra-hour peaks. Stochastic variables (wind/solar) use linear interpolation.
• Meteo Data: Utilizes lapse-rate aware k-Nearest Neighbors (k-NN) to prioritize vertical similarity, and Time-of-Day (TOD) fallbacks to bridge gaps when spatial neighbors are unavailable.

Stage 2: Statistical Reconstruction (Imputation Phase)

Targets structural outages and historical backfilling for gaps > 1h.

• TS-ICL Engine (Default): A zero-shot time-series foundation model. It handles raw temporal context directly and ignores the training_cutoff_years parameter.
• XGBoost Engine (Fallback): An extreme gradient boosting regressor that uses engineered lag features, thermal inertia smoothing, and a Brownian bridge to ensure $C_0$ continuity at gap boundaries.

---

Dataset Output Structure

The Outputs/Imputed/ directory contains the fully reconstructed datasets in two frequencies (15min, 30min).

Primary Files

• dataset_15min_france_load.csv: Drivers for consumption forecasting (temperature, nebulosity, calendar).
• dataset_15min_france_wind.csv: Drivers for wind power (wind speed, air density, gusts).
• dataset_15min_france_solar.csv: Drivers for photovoltaics (irradiance proxies, temperature).
• dataset_15min_full.csv: The complete high-dimensional matrix (~400 columns).

Key Feature Glossary

Feature	Unit	Description
rte_load_france	MW	Reference record for national load (best of definitive/consolidated/real-time).
rte_load_france_definitive	MW	Strictly validated historical load (isolated for deviation/error analysis).
enedis_load_residential_total_france	MW	Reconstructed residential load combining telemetered (Linky) and profiled data.
meteo_temperature_celsius_france_load	°C	National temperature proxy, dynamically weighted by regional population density.
day_type_week_period_hour_changed	Cat	Composite calendar key encoding day of the week, holidays, and seasonality.

---

Project Structure

force/
├── src/force/
│   ├── collectors/       # API Connectors (State-Aware & Robust)
│   ├── transformation/   # Physics-Aware Smoothing & Multi-Version Pivoting
│   ├── check_*.py        # Quality Control & Auditing Modules
│   └── imputation/       # ML Imputation Engine (Predictive, Spatial, Audit)
├── documentation/        # Detailed Methodological Papers
├── data_warehouse/       # Local DuckDB Storage (Normalized Schemas)
└── Outputs/              # Final CSV Datasets

---

Dataset Access

Recognizing that reproducing the full ETL pipeline requires time, computational resources and reliable API access, we provide a static, versioned snapshot of the finalized dataset hosted on Zenodo.

• Download the 2026 Dataset Snapshot: https://doi.org/10.5281/zenodo.21109134

---

Citation

If you use the FORCE package or the pre-compiled dataset in your research, please cite them using their permanent archives:

To cite the FORCE Software Pipeline:

@misc{force2026package,
  title={FORCE: French Open Research Catalogue of Energy (v1.1.1)},
  author={Allioux, Yann and Goude, Yannig},
  year={2026},
  doi={10.5281/zenodo.21108937},
  note={With a contribution of Nathan Doumèche during his thesis}
}

To cite the FORCE Dataset Snapshot:

@dataset{force2026dataset,
  title={FORCE Dataset: French Open Research Catalogue of Energy - 2026 Snapshot},
  author={Allioux, Yann and Goude, Yannig},
  year={2026},
  publisher={Zenodo},
  doi={10.5281/zenodo.21109134},
  url={https://doi.org/10.5281/zenodo.21109134}
}

---

License & Copyright

FORCE: French Open Research Catalogue of Energy is released under the LGPL-3.0-or-later license.

• 2025-2026 Advanced imputation engine, pipeline architecture, Enedis & radiation features, DuckDB Ecosystem: Copyright © EDF, author Yann Allioux.
• 2023 Imputation engine & scientific publication: Copyright © EDF & Sorbonne Université, authors Yann Allioux, Nathan Doumèche.
• 2020-2022 Core framework & baselines: Copyright © EDF, author Yann Allioux.

Data Attribution & Licenses:

• Grid and synoptic weather data provided by RTE, Enedis, Météo-France, and ODRÉ under the Etalab Open License 2.0.
• Solar radiation baseline data provided by the European Commission PVGIS (SARAH-3).
• Reanalysis and climate projection data generated using Copernicus Climate Change Service information (ERA5, CMIP6) and subject to the Copernicus License. Users of this pipeline must register for a Copernicus account and accept their Terms of Use.