Skip to main content

Nowcasting Dataset

Project description



All Contributors


Pre-prepare batches of data for use in machine learning training.

This code combines several data sources including:

  • Satellite imagery (EUMETSAT SEVIRI RSS 5-minutely data of UK)
  • Numerical Weather Predictions (NWPs. UK Met Office UKV model from CEDA)
  • Solar PV power timeseries data (from, downloaded using our pvoutput Python code.)
  • Estimated total solar PV generation for each of the ~350 "grid supply points" (GSPs) in Britain from Sheffield Solar's PV Live Regional API.
  • Topographic data.
  • The Sun's azimuth and angle.

This repo doesn't contain the ML models themselves. Please see this page for an overview of the Open Climate Fix solar PV nowcasting project, and how our code repositories fit together.

User manual



From within the cloned nowcasting_dataset directory:

conda env create -f environment.yml
conda activate nowcasting_dataset
pip install -e .


A (probably older) version is also available through pip install nowcasting-dataset


If you want to also install PVLive then use pip install git+


A pre commit hook has been installed which makes black run with every commit. You need to install black and pre-commit (these will be installed by conda or pip when installing nowcasting_dataset) and run pre-commit install in this repo.


To test using the small amount of data stored in this repo: py.test -s

To output debug logs while running the tests then run py.test --log-cli-level=10

To test using the full dataset on Google Cloud, add the --use_cloud_data switch.


Test using a docker file and database

docker stop $(docker ps -a -q)
docker-compose -f test-docker-compose.yml build
docker-compose -f test-docker-compose.yml run dataset

Downloading data

Satellite data

Use Satip to download native EUMETSAT SEVIRI RSS data from EUMETSAT's API and then convert to an intermediate file format.

PV data from

Download PV timeseries data from using our PVOutput code.

OCF uk_pv dataset

PV solar generation data from the UK. This dataset contains data from 1311 PV systems from 2018-01-01 to 2021-10-27. The time series of solar generation is in 5 minutes chunks. This data is collected from live PV systems in the UK. We have obfuscated the location of the PV systems for privacy.

Numerical weather predictions from the UK Met Office

Please use our nwp code to download UKV NWPs and convert to Zarr.

GSP-level estimates of PV outturn from PV Live Regional


Topographical data

  1. Make an account at the USGS EarthExplorer website
  2. Create a region of the world to download data for, in our case, the spatial extant of the SEVIRI RSS image
  3. Select the data products you want, in this case SRTM elevation maps
  4. Download all the SRTM files that cover that area

There does not seem to be an automated way to do this selecting and downloading, so this might take awhile.

Configure nowcasting_dataset to point to the downloaded data

Copy and modify one of the config yaml files in nowcasting_dataset/config/.

Prepare ML batches

Run scripts/ --help to learn how to run the script.

What exactly is in each batch?

Please see the data_sources/<modality>/<modality> files (where <modality> is one of {datetime, metadata, gsp, nwp, pv, satellite, sun, topographic}) for documentation about the different data fields in each example / batch.

History of nowcasting_dataset

When we first started writing nowcasting_dataset, our intention was to load and align data from these three datasets on-the-fly during ML training. But it just isn't quite fast enough to keep a modern GPU constantly fed with data when loading multiple satellite channels and multiple NWP parameters. So, now, this code is used to pre-prepare thousands of batches, and save these batches to disk, each as a separate NetCDF file. These files can then be loaded super-quickly at training time. The end result is a 12x speedup in training.

Contributors ✨

Thanks goes to these wonderful people (emoji key):

Jack Kelly


Jacob Bieker


Peter Dudfield




Rohan Nuttall


Nasser Benabderrazik


Shanmukh Chava


Rishi Kumar Ray




This project follows the all-contributors specification. Contributions of any kind welcome!

Project details

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

nowcasting_dataset-3.8.2.tar.gz (103.8 kB view hashes)

Uploaded source

Built Distribution

nowcasting_dataset-3.8.2-py3-none-any.whl (128.9 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page