Skip to main content

PVNet

All Contributors

tags badge ease of contribution: hard

This project is used for training PVNet and running PVNet on live data.

PVNet is a multi-modal late-fusion model for predicting renewable energy generation from weather data. The NWP (Numerical Weather Prediction) and satellite data are sent through a neural network which encodes them down to 1D intermediate representations. These are concatenated together with recent generation, the calculated solar coordinates (azimuth and elevation) and the location ID which has been put through an embedding layer. This 1D concatenated feature vector is put through an output network which outputs predictions of the future energy yield.

Experiments

Our paper based on this repo was accepted into the Tackling Climate Change with Machine Learning workshop at ICLR 2024 and can be viewed here.

Some more structured notes on experiments we have performed with PVNet are here.

Setup / Installation

git clone git@github.com:openclimatefix/PVNet.git
cd PVNet
pip install .

The commit history is extensive. To save download time, use a depth of 1:

git clone --depth 1 git@github.com:openclimatefix/PVNet.git

This means only the latest commit and its associated files will be downloaded.

Next, in the PVNet repo, install PVNet as an editable package:

pip install -e .

Additional development dependencies

pip install ".[dev]"

Getting started with running PVNet

Before running any code in PVNet, copy the example configuration to a configs directory:

cp -r configs.example configs

You will be making local amendments to these configs. See the README in configs.example for more info.

Datasets

As a minimum, in order to create samples of data/run PVNet, you will need to supply paths to NWP and GSP data. PV data can also be used. We list some suggested locations for downloading such datasets below:

GSP (Grid Supply Point) - Regional PV generation data
The University of Sheffield provides API access to download this data: https://www.solar.sheffield.ac.uk/api/

Documentation for querying generation data aggregated by GSP region can be found here: https://docs.google.com/document/d/e/2PACX-1vSDFb-6dJ2kIFZnsl-pBQvcH4inNQCA4lYL9cwo80bEHQeTK8fONLOgDf6Wm4ze_fxonqK3EVBVoAIz/pub#h.9d97iox3wzmd

NWP (Numerical weather predictions)
OCF maintains a Zarr formatted version of the German Weather Service's (DWD) ICON-EU NWP model here: https://huggingface.co/datasets/openclimatefix/dwd-icon-eu which includes the UK

PV
OCF maintains a dataset of PV generation from 1311 private PV installations here: https://huggingface.co/datasets/openclimatefix/uk_pv

Connecting with ocf-data-sampler for sample creation

Outside the PVNet repo, clone the ocf-data-sampler repo and exit the conda env created for PVNet: https://github.com/openclimatefix/ocf-data-sampler

git clone git@github.com/openclimatefix/ocf-data-sampler.git
conda create -n ocf-data-sampler python=3.11

Then go inside the ocf-data-sampler repo to add packages

pip install .

Then exit this environment, and enter back into the pvnet conda environment and install ocf-data-sampler in editable mode (-e). This means the package is directly linked to the source code in the ocf-data-sampler repo.

pip install -e <PATH-TO-ocf-data-sampler-REPO>

If you install the local version of ocf-data-sampler that is more recent than the version specified in PVNet it is not guarenteed to function properly with this library.

Set up and config example for streaming

We will use the following example config file to describe your data sources: /PVNet/configs/datamodule/configuration/example_configuration.yaml. Ensure that the file paths are set to the correct locations in example_configuration.yaml: search for PLACEHOLDER to find where to input the location of the files. Delete or comment the parts for data you are not using.

At run time, the datamodule config PVNet/configs/datamodule/streamed_samples.yaml points to your chosen configuration file:

configuration: "/FULL-PATH-TO-REPO/PVNet/configs/datamodule/configuration/example_configuration.yaml"

You can also update train/val/test time ranges here to match the period you have access to.

If downloading private data from a GCP bucket make sure to authenticate gcloud (the public satellite data does not need authentication):

gcloud auth login

You can provide multiple storage locations as a list. For example:

satellite: zarr_path: - "gs://public-datasets-eumetsat-solar-forecasting/satellite/EUMETSAT/SEVIRI_RSS/v4/2020_nonhrv.zarr" - "gs://public-datasets-eumetsat-solar-forecasting/satellite/EUMETSAT/SEVIRI_RSS/v4/2021_nonhrv.zarr"

ocf-data-sampler is currently set up to use 11 channels from the satellite data (the 12th, HRV, is not used).

⚠️ NB: Our publicly accessible satellite data is currently saved with a blosc2 compressor, which is not supported by the tensorstore backend PVNet relies on now. We are in the process of updating this; for now, the paths above cannot be used with this codebase.

Training PVNet

How PVNet is run is determined by the configuration files. The example configs in PVNet/configs.example work with streamed_samples using datamodule/streamed_samples.yaml.

Update the following before training:

  1. In configs/model/late_fusion.yaml:
    • Update the list of encoders to match the data sources you are using. For different NWP sources, keep the same structure but ensure:
      • in_channels: the number of variables your NWP source supplies
      • image_size_pixels: spatial crop matching your NWP resolution and the settings in your datamodule configuration (unless you coarsened, e.g. for ECMWF)
  2. In configs/trainer/default.yaml:
    • Set accelerator: 0 if running on a system without a supported GPU
  3. In configs/datamodule/streamed_samples.yaml:
    • Point configuration: to your local example_configuration.yaml (or your custom one)
    • Adjust the train/val/test time ranges to your available data

If you create custom config files, update the main ./configs/config.yaml defaults:

defaults:

  • trainer: default.yaml
  • model: late_fusion.yaml
  • datamodule: streamed_samples.yaml
  • callbacks: null
  • experiment: null
  • hparams_search: null
  • hydra: default.yaml

Now train PVNet:

python run.py

You can override any setting with Hydra, e.g.:

python run.py datamodule=streamed_samples datamodule.configuration="/FULL-PATH/PVNet/configs/datamodule/configuration/example_configuration.yaml"

Evaluation

After training, you can evaluate model performance on the validation dataset, for example using scripts/mae_analysis.py. Once trained, you can push the model checkpoints to hugging face with scripts/checkpoint_to_huggingface.py. This script is also used to store the best checkpoint locally with the flag --no-push-to-hub.

If you have successfully trained a PVNet model and have a saved model checkpoint, you can find instructions on how to run a backtest and evaluate it in ml-tooling (for internal OCF use only).

Testing

You can use python -m pytest tests to run tests

Contributors ✨

Thanks goes to these wonderful people (emoji key):

Felix
Felix

💻
Sukhil Patel
Sukhil Patel

💻
James Fulton
James Fulton

💻
Alexandra Udaltsova
Alexandra Udaltsova

💻 👀
Megawattz
Megawattz

💻
Peter Dudfield
Peter Dudfield

💻
Mahdi Lamb
Mahdi Lamb

🚇
Jacob Prince-Bieker
Jacob Prince-Bieker

💻
codderrrrr
codderrrrr

💻
Chris Briggs
Chris Briggs

💻
tmi
tmi

💻
Chris Arderne
Chris Arderne

💻
Dakshbir
Dakshbir

💻
MAYANK SHARMA
MAYANK SHARMA

💻
aryan lamba
aryan lamba

💻
michael-gendy
michael-gendy

💻
Aditya Suthar
Aditya Suthar

💻
Markus Kreft
Markus Kreft

💻
Jack Kelly
Jack Kelly

🤔
zaryab-ali
zaryab-ali

💻
Lex-Ashu
Lex-Ashu

💻

This project follows the all-contributors specification. Contributions of any kind welcome!

Metadata

Release files for PVNet 5.3.27

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for PVNet 5.3.27
File Size Uploaded Provenance Actions
pvnet-5.3.27.tar.gz 35.4 kB Without provenance Details for pvnet-5.3.27.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for PVNet 5.3.27
File Interpreter ABI Platform Actions
pvnet-5.3.27-py3-none-any.whl Python 3 none any Details for pvnet-5.3.27-py3-none-any.whl

Total release size: 77.8 kB

Release files / pvnet-5.3.27.tar.gz

Download URL pvnet-5.3.27.tar.gz
Size 35.4 kB
Tags Source
SHA-256 checksum
How to use checksums
6fd6194eca28854c9ab3501a89514b98d852a93b4237b8139aeefd3919ae75d7
BLAKE2b-256 checksum
How to use checksums
ebceea3aefe5421c7ed72d7b9f8f88a29d2fe45d09d9383458f27381914c2eb4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / pvnet-5.3.27-py3-none-any.whl

Download URL pvnet-5.3.27-py3-none-any.whl
Size 42.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
02aa32fb59440892a7e399828d2728b494576df1af939dafdd1061505726d555
BLAKE2b-256 checksum
How to use checksums
a4d7036e8d19e40660a2c0f0314d1f7a08b6f59db73e4b4ebc2f14bb8f1f3eaa
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for PVNet 5.3.27

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

5.3.27 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page