Nowcasting Dataset
Project description
nowcasting_dataset
Pre-prepare batches of data for use in machine learning training.
This code combines several data sources including:
- Satellite imagery (EUMETSAT SEVIRI RSS 5-minutely data of UK)
- Numerical Weather Predictions (NWPs. UK Met Office UKV model from CEDA)
- Solar PV power timeseries data (from PVOutput.org, downloaded using our pvoutput Python code.)
- Estimated total solar PV generation for each of the ~350 "grid supply points" (GSPs) in Britain from Sheffield Solar's PV Live Regional API.
- Topographic data.
- The Sun's azimuth and angle.
This repo doesn't contain the ML models themselves. Please see this page for an overview of the Open Climate Fix solar PV nowcasting project, and how our code repositories fit together.
User manual
Installation
conda
From within the cloned nowcasting_dataset
directory:
conda env create -f environment.yml
conda activate nowcasting_dataset
pip install -e .
pip
A (probably older) version is also available through pip install nowcasting-dataset
PV Live API
If you want to also install PVLive then use pip install git+https://github.com/SheffieldSolar/PV_Live-API
Pre-commit
A pre commit hook has been installed which makes black
run with every commit. You need to install
black
and pre-commit
(these will be installed by conda
or pip
when installing
nowcasting_dataset
) and run pre-commit install
in this repo.
Testing
To test using the small amount of data stored in this repo: py.test -s
To test using the full dataset on Google Cloud, add the --use_cloud_data
switch.
Downloading data
Satellite data
Use Satip to download native EUMETSAT SEVIRI RSS data from EUMETSAT's API and then convert to an intermediate file format.
PV data from PVOutput.org
Download PV timeseries data from PVOutput.org using our PVOutput code.
Numerical weather predictions from the UK Met Office
Request access to the UK Met Office data on CEDA.
Once you have a username and password, download using
scripts/download_UK_Met_Office_NWPs_from_CEDA.sh
.
Please see the comments at the top of the script for instructions.
Detailed docs of the Met Office data is available here.
GSP-level estimates of PV outturn from PV Live Regional
TODO
Topographical data
- Make an account at the USGS EarthExplorer website
- Create a region of the world to download data for, in our case, the spatial extant of the SEVIRI RSS image
- Select the data products you want, in this case SRTM elevation maps
- Download all the SRTM files that cover that area
There does not seem to be an automated way to do this selecting and downloading, so this might take awhile.
Configure nowcasting_dataset
to point to the downloaded data
Copy and modify one of the config yaml files in
nowcasting_dataset/config/
.
Prepare ML batches
Run scripts/prepare_ml_data.py --help
to learn how to run the prepare_ml_data.py
script.
What exactly is in each batch?
Please see the data_sources/<modality>/<modality>_model.py
files
(where <modality>
is one of {datetime, metadata, gsp, nwp, pv,
satellite, sun, topographic}) for documentation about the different
data fields in each example / batch.
History of nowcasting_dataset
When we first started writing nowcasting_dataset
, our intention was
to load and align data from these three datasets on-the-fly during ML
training. But it just isn't quite fast enough to keep a modern GPU constantly fed
with data when loading multiple satellite channels and multiple NWP
parameters. So, now, this code is used to pre-prepare thousands of
batches, and save these batches to disk, each as a separate NetCDF
file. These files can then be loaded super-quickly at training time.
The end result is a 12x speedup in training.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for nowcasting_dataset-2.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 15d01726d246ef6df9702ad2fcf90bfb555e218ebbee7930827439ef37e1e799 |
|
MD5 | f2a8ed93812d720fb1e808614e9954a3 |
|
BLAKE2b-256 | d152785886ebe40a6d17bf43d1d2aa0408d10d3f20fbba702d16e6c9933ba174 |