era5-s3-dl
CLI tool for downloading ERA5 reanalysis data from NCAR's AWS S3 archive, clipping files to geographic bounds, and uploading the results to a remote destination. As of this writing, the total size of the ERA5 global files is 236 TB -- you will definitely want to clip to your region.
For each file, the tool: downloads from the NCAR source via rclone, clips to lat/lon bounds via ncks (converting to NetCDF-4 with compression), and uploads the clipped file to a configured remote destination.
Prerequisites
- Python >= 3.10
- ncks (part of NCO) must be installed separately for NetCDF clipping
Installation
pipx install era5_s3_dl
With optional Sentry error tracking (used in Docker deployment):
pipx install era5_s3_dl[sentry]
Docker
The Docker image includes ncks and installs the tool with the Sentry extra. Modify parameters.toml and docker-compose.yml, then run:
docker-compose up -d
docker-compose logs -f
The docker-compose.yml mounts your parameters.toml into the container and maps an output volume:
volumes:
- "./parameters.toml:/parameters.toml"
- "./output:/data/output"
Configuration
Copy parameters_example.toml to parameters.toml and edit it. The TOML file is the primary configuration mechanism -- all settings live here, and select values can be overridden via CLI options.
TOML sections
Root-level settings:
| Key | Type | Default | Description |
|---|---|---|---|
n_tasks |
int | 8 |
Number of parallel download workers |
check_target |
bool | true |
Check remote for existing files before downloading (skips duplicates) |
work_dir |
string | (temp dir) | Working directory for intermediate files. Creates download/ and clipped/ subdirs within it. If not set, uses a temporary directory that is cleaned up on completion. |
preset |
string | Variable preset (e.g. "wrf", "all"). Required unless variables is set. |
|
variables |
string | Comma-separated NetCDF variable names to download (case insensitive). Combined with preset if both set. |
|
skip_vars |
string | Comma-separated NetCDF variable names to exclude (case insensitive). Applied after preset + variables. |
[dates] -- Date range for files to download:
[dates]
start_date = "2020-01-01"
end_date = "2024-12-31"
[bounds] -- Geographic clipping bounds (longitude/latitude):
[bounds]
min_lon = 145
max_lon = 195
min_lat = -60
max_lat = -20
[source] -- rclone config for the NCAR ERA5 source bucket. The default points to the public nsf-ncar-era5 bucket on AWS:
[source]
type = 's3'
provider = 'AWS'
env_auth = 'false'
region = 'us-west-2'
path = 'nsf-ncar-era5/'
[remote] -- rclone config for the upload destination. Can be S3 or a local path. Optional for --list-only mode:
# S3 example:
[remote]
type = 's3'
provider = 'Mega'
endpoint = 'https://s3.example.com'
access_key_id = ''
secret_access_key = ''
path = '/data/ncar/era5/'
# Local example (use with Docker volume mount):
[remote]
type = 'local'
path = '/data/output'
[sentry] (optional) -- Sentry error tracking. Requires the sentry extra to be installed:
[sentry]
dsn = ''
tags = {}
Variable Selection
You must specify which ERA5 variables to download using --preset and/or --variables.
Presets select a predefined group of variables:
wrf-- the 26 variables required for WRF model initializationall-- all 92 available ERA5 variables
Variables are specified by their NetCDF variable name (case insensitive). The full catalog is in era5_dl/era5_variables.json. Examples: SP, VAR_2T, Z, CAPE, U, V.
When both --preset and --variables are given, the variables are combined (union).
Skip-vars (--skip-vars) removes variables after --preset and --variables are resolved. Useful when you want a preset minus a few fields — e.g. WRF inputs but with SST and sea ice coming from a different dataset.
# Download WRF variables only
era5_dl parameters.toml --preset wrf
# Download just surface pressure and 2m temperature
era5_dl parameters.toml --variables sp,var_2t
# Download WRF variables plus CAPE
era5_dl parameters.toml --preset wrf --variables cape
# Download WRF variables but skip SST and sea ice (sourcing them elsewhere)
era5_dl parameters.toml --preset wrf --skip-vars sstk,ci
# List all available files without downloading
era5_dl parameters.toml --preset all --list-only
Usage
Basic invocation with a parameters file:
era5_dl parameters.toml --preset wrf
Override specific TOML values via CLI options:
era5_dl parameters.toml --preset wrf -s 2024-01-01 -e 2024-12-31 -n 4
era5_dl parameters.toml --preset wrf --min-lon 160 --max-lon 180 --no-check-target
List matching files without downloading (does not require [remote] config):
era5_dl parameters.toml --preset wrf --list-only
See all options:
era5_dl --help
CLI Options
All options are optional and override the corresponding value in the parameters TOML file.
| Option | Short | Type | Overrides |
|---|---|---|---|
--n-tasks |
-n |
int | n_tasks |
--work-dir |
-w |
path | work_dir |
--check-target / --no-check-target |
bool | check_target |
|
--start-date |
-s |
text | dates.start_date |
--end-date |
-e |
text | dates.end_date |
--min-lon |
float | bounds.min_lon |
|
--max-lon |
float | bounds.max_lon |
|
--min-lat |
float | bounds.min_lat |
|
--max-lat |
float | bounds.max_lat |
|
--preset |
-p |
text | preset |
--variables |
-v |
text | variables |
--skip-vars |
text | skip_vars |
|
--list-only |
-l |
flag | Query and list files only, skip download |
Exit status and messages
Since 0.2.2, era5_dl exits 1 if any file failed at any step, or if listing a source directory failed, so a
caller can trust exit 0 to mean every selected file reached the target. Each step (download, clip, upload) is judged by
its tool's return code, not by whether the tool wrote to stderr: rclone writes to stderr on success too, for example
ERROR : Attempt 1/3 failed before a retry succeeds. Before 0.2.2 such a recovered transfer counted as a failure, the
good file was deleted, and the run still exited 0.
NOTICE <step> <key>: <message>(stdout): the step succeeded but its tool wrote to stderr. The file is kept.FAILED <step> <key> rc=<code>: <message>(stderr): the file did not reach the target.-- <n> of <m> files FAILED(stderr), then exit 1.
--check-target (the default) lists the target with rclone lsf and downloads every selected file it does not hold.
A target that cannot be listed is an error (exit 1). Before 0.2.2 this used rclone check --missing-on-dst, which
exits 1 for an error as for a difference, and its empty output on an error was read as the key '', a copy of the
whole source root.
Metadata
Release files for era5-s3-dl 0.2.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| era5_s3_dl-0.2.2.tar.gz | 18.6 kB | Without provenance | Details for era5_s3_dl-0.2.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| era5_s3_dl-0.2.2-py3-none-any.whl | Python 3 | none | any | Details for era5_s3_dl-0.2.2-py3-none-any.whl |
Total release size: 36.3 kB
Release files / era5_s3_dl-0.2.2.tar.gz
| Download URL | era5_s3_dl-0.2.2.tar.gz |
|---|---|
| Size | 18.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
517c68f6db1a6ea93abe6a4fa5a840bb7d60fd0cc4d2437aa0a4ba6cc7f8879c
|
|
BLAKE2b-256 checksum How to use checksums |
02cc2b3dc1ce5d159fae551268b64817adb44d3c7bb90ae1ef964e136d33f802
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.8.7
|
Release files / era5_s3_dl-0.2.2-py3-none-any.whl
| Download URL | era5_s3_dl-0.2.2-py3-none-any.whl |
|---|---|
| Size | 17.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
034fec5cbc4b6a0379fdbd00bc95563113b110ee8c995a4b3e52781d44e237a8
|
|
BLAKE2b-256 checksum How to use checksums |
e093187c861d8b891f87be08e15d6961c0a8ac9658f608cf4e0004f07f35001b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.8.7
|
Security for era5-s3-dl 0.2.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.