Skip to main content

a simple helper for downloading ECMWF's ERA5 reanalysis data

Project description

ERA5-dl: a simple helper for downloading ECMWF's ERA5 reanalysis data

Dependencies and requirements:

Install

Install via pip:

pip install era5dl

Features and usages

1. Batch download

Send batch download jobs to retrieve large amount of data while saving the downloaded data into separate files, e.g. each for a year.

E.g. to download u-wind and geo-potential during 2000-2001, on pressure levels 1000 and 800 hPa, while skipping some combinations of variables, years and levels:

from era5dl import batchDownload, TEMPLATE_DICT

OUTPUTDIR='.'

JOB_DICT = {
    'variable': ['u_component_of_wind', 'geopotential'],
    'year': range(2000, 2002),
    'pressure_level': [1000, 800]
}

SKIP_LIST = [
    {'variable': 'u_component_of_wind', 'year': [2000, ], 'pressure_level': [1000, 800]},
    {'variable': 'geopotential', 'year': [2001, ], 'pressure_level': [800, ]}, ]

batchDownload(TEMPLATE_DICT, JOB_DICT, SKIP_LIST, OUTPUTDIR, dry=True, pause=3)

2. Keep a log

A log file is created in the same folder where the downloaded data are saved.

Example log:

<util_downloader.py-processJob()>: 2021-04-17 20:20:38,289,INFO: <batch_download>: Output folder at: ./
<util_downloader.py-processJob()>: 2021-04-17 20:20:38,290,INFO: Launch job 1
<util_downloader.py-processJob()>: 2021-04-17 20:20:38,290,INFO: Job info: {'product_type': 'reanalysis', 'format': 'netcdf', 'variable': 'u_component_of_wind', 'pressure_level': 800, 'year': 2001, 'month': ['01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12'], 'day': ['01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31'], 'time': ['00:00', '06:00', '12:00', '18:00'], 'area': [10, 80, -10, 100]}
<util_downloader.py-processJob()>: 2021-04-17 20:20:38,290,INFO: Output file location: ./[ID0]800-u_component_of_wind-2001.nc
<util_downloader.py-processJob()>: 2021-04-17 20:20:41,293,INFO: <batch_download>: Output folder at: ./
<util_downloader.py-processJob()>: 2021-04-17 20:20:41,294,INFO: Launch job 2
<util_downloader.py-processJob()>: 2021-04-17 20:20:41,294,INFO: Job info: {'product_type': 'reanalysis', 'format': 'netcdf', 'variable': 'u_component_of_wind', 'pressure_level': 1000, 'year': 2001, 'month': ['01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12'], 'day': ['01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31'], 'time': ['00:00', '06:00', '12:00', '18:00'], 'area': [10, 80, -10, 100]}
<util_downloader.py-processJob()>: 2021-04-17 20:20:41,294,INFO: Output file location: ./[ID1]1000-u_component_of_wind-2001.nc
<util_downloader.py-processJob()>: 2021-04-17 20:20:44,296,INFO: <batch_download>: Output folder at: ./
<util_downloader.py-processJob()>: 2021-04-17 20:20:44,296,INFO: Launch job 3
...

3. Skip already downloaded files

When running a batch downloading job, each finished job is recorded in a text file named downloaded_list.txt in the same folder as the saved data. If the downloading is interrupted, for instance by network issues, a second run of the script will first look at the downloaded_list.txt file and exclude those already finished retrievals.

4. Create a batch download job by splitting the api request from ECMWF web

E.g.

One selects the desired data from the CDS web interface as shown in the following 3 screen captures:

Notice that it is warned that the requested field is too large. Even if not, one may want to split the entire data into smaller, more manageable chunks, for instance, by saving each variable in each year, on each vertical level into a separate file.

To split the retrieval, first click the Show API request button at the bottom of the page, and copy and save the Python code into a text file, e.g. api.txt, then run a Python script with the following content:

from era5dl import batchDownloadFromWebRequest

OUTPUTDIR='./'
DRY=False

batchDownloadFromWebRequest('./api.txt', OUTPUTDIR,
    ['variable', 'pressure_level', 'year'], DRY, pause=3)

The ['variable', 'pressure_level', 'year'] list tells that the batch job is split by these 3 dimensions/fields, such that each sub-job consists of each variable in each year, on each vertical level, and the data of the sub-job is saved into a separate file.

Again, already downloaded data are recorded in the downloaded_list.txt file and re-executing the script will not re-download them.

5. Automatically generate meaningful file names

The batchDownload() and batchDownloadFromWebRequest() functions accept a naming_func keyword argument, which can be None, or a callable.

If a callable, it should be a function that accepts a single input argument which is a dict defining the data retrieval task, and returns a string as the filename (without folder path) to name the downloaded data.

If None, it will construct a default filename, using the following format:

[ID<n>]<attributes>.nc or [ID<n>]<attributes>.grb.

where <n> is the numerical id of the job, <attributes> is a dash concatenated string joining the attributes that define the job. E.g.

[ID02]700-geopotential-2000.nc

6. Dry run

The batchDownload() and batchDownloadFromWebRequest() functions accept a dry positional argument. When set to True, will simulate the retrieval rather than actually sending the cdsapi retrieval request. This can be used to test the request definition.

Contribution

This tool is still in early development stage. Contributions and bug reports are welcome. Please create a fork of the project on GitHub and use a pull request to propose your changes.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for era5dl, version 0.1a3
Filename, size File type Python version Upload date Hashes
Filename, size era5dl-0.1a3-py3-none-any.whl (42.2 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size era5dl-0.1a3.tar.gz (37.1 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page