Nowcasting Dataset
Project description
nowcasting_dataset
A multi-process data loader for PyTorch, optimised for Google Cloud, which aligns three separate datasets:
- Satellite imagery (EUMETSAT SEVIRI RSS 5-minutely data of UK)
- Numerical Weather Predictions (NWPS. UK Met Office UKV model)
- Solar PV power timeseries data (from PVOutput.org, downloaded using our pvoutput Python code.)
At the start of writing this code, the intention was to load and align data from these three datasets on-the-fly during ML training. And it can still be used that way! But it just isn't quite fast enough to keep a modern GPU constantly fed with data when loading multiple satellite channels and multiple NWP parameters. So, now, this code is used to pre-prepare thousands of batches, and save these batches to disk, each as a separate NetCDF file. These files can then be loaded super-quickly at training time. The end result is a 12x speedup in training.
The script scripts/prepare_ml_training_data.py
is used to
pre-compute the training and validation data (the script makes used of the
nowcasting_dataset
library).
nowcasting_dataset.dataset.datasets.NetCDFDataset
is a PyTorch Dataset, to
be used to train ML models.
This repo doesn't contain the ML models themselves. The models are in: https://github.com/openclimatefix/predict_pv_yield/ and https://github.com/openclimatefix/satflow, and utils are in https://github.com/openclimatefix/nowcasting_utils
Installation
From within the cloned nowcasting_dataset
directory:
conda env create -f environment.yml
conda activate nowcasting_dataset
pip install -e .
sudo apt install libgl1-mesa-glx # For optical flow
A (probably older) version is also available through pip install nowcasting-dataset
pip
You can install a local version with pip install -e .
.
If you want to also install PVLive then use pip install git+https://github.com/SheffieldSolar/PV_Live-API
Pre-commit
A pre commit hook has been installed which makes black
run with every commit. You need to install
black
and pre-commit
(these will be installed by conda
or pip
when installing
nowcasting_dataset
) and run pre-commit install
in this repo.
Testing
To test using the small amount of data stored in this repo: py.test -s
To test using the full dataset on Google Cloud, add the --use_cloud_data
switch.
Documentation
Please see the Example
class for documentation about the different data fields in each example / batch.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for nowcasting_dataset-0.1.5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 30d9113c58d994babf4fdf4756d1e09db26fe097d64254605851b053dcd27a7a |
|
MD5 | 5546288fa8791d83c36018330dac27b8 |
|
BLAKE2b-256 | 450a0f8c5c34f834a64593fc898945b88c8ed9352d0805bfa3074b16d0e79d23 |