Skip to main content

era5-s3-dl

CLI tool for downloading ERA5 reanalysis data from NCAR's AWS S3 archive, clipping files to geographic bounds, and uploading the results to a remote destination. As of this writing, the total size of the ERA5 global files is 236 TB -- you will definitely want to clip to your region.

For each file, the tool: downloads from the NCAR source via rclone, clips to lat/lon bounds via ncks (converting to NetCDF-4 with compression), and uploads the clipped file to a configured remote destination.

Prerequisites

  • Python >= 3.10
  • ncks (part of NCO) must be installed separately for NetCDF clipping

Installation

pipx install era5_s3_dl

With optional Sentry error tracking (used in Docker deployment):

pipx install era5_s3_dl[sentry]

Docker

The Docker image includes ncks and installs the tool with the Sentry extra. Modify parameters.toml and docker-compose.yml, then run:

docker-compose up -d
docker-compose logs -f

The docker-compose.yml mounts your parameters.toml into the container and maps an output volume:

volumes:
    - "./parameters.toml:/parameters.toml"
    - "./output:/data/output"

Configuration

Copy parameters_example.toml to parameters.toml and edit it. The TOML file is the primary configuration mechanism -- all settings live here, and select values can be overridden via CLI options.

TOML sections

Root-level settings:

Key Type Default Description
n_tasks int 8 Number of parallel download workers
check_target bool true Check remote for existing files before downloading (skips duplicates)
work_dir string (temp dir) Working directory for intermediate files. Creates download/ and clipped/ subdirs within it. If not set, uses a temporary directory that is cleaned up on completion.
preset string Variable preset (e.g. "wrf", "all"). Required unless variables is set.
variables string Comma-separated NetCDF variable names to download (case insensitive). Combined with preset if both set.
skip_vars string Comma-separated NetCDF variable names to exclude (case insensitive). Applied after preset + variables.

[dates] -- Date range for files to download:

[dates]
start_date = "2020-01-01"
end_date = "2024-12-31"

[bounds] -- Geographic clipping bounds (longitude/latitude):

[bounds]
min_lon = 145
max_lon = 195
min_lat = -60
max_lat = -20

[source] -- rclone config for the NCAR ERA5 source bucket. The default points to the public nsf-ncar-era5 bucket on AWS:

[source]
type = 's3'
provider = 'AWS'
env_auth = 'false'
region = 'us-west-2'
path = 'nsf-ncar-era5/'

[remote] -- rclone config for the upload destination. Can be S3 or a local path. Optional for --list-only mode:

# S3 example:
[remote]
type = 's3'
provider = 'Mega'
endpoint = 'https://s3.example.com'
access_key_id = ''
secret_access_key = ''
path = '/data/ncar/era5/'

# Local example (use with Docker volume mount):
[remote]
type = 'local'
path = '/data/output'

[sentry] (optional) -- Sentry error tracking. Requires the sentry extra to be installed:

[sentry]
dsn = ''
tags = {}

Variable Selection

You must specify which ERA5 variables to download using --preset and/or --variables.

Presets select a predefined group of variables:

  • wrf -- the 26 variables required for WRF model initialization
  • all -- all 92 available ERA5 variables

Variables are specified by their NetCDF variable name (case insensitive). The full catalog is in era5_dl/era5_variables.json. Examples: SP, VAR_2T, Z, CAPE, U, V.

When both --preset and --variables are given, the variables are combined (union).

Skip-vars (--skip-vars) removes variables after --preset and --variables are resolved. Useful when you want a preset minus a few fields — e.g. WRF inputs but with SST and sea ice coming from a different dataset.

# Download WRF variables only
era5_dl parameters.toml --preset wrf

# Download just surface pressure and 2m temperature
era5_dl parameters.toml --variables sp,var_2t

# Download WRF variables plus CAPE
era5_dl parameters.toml --preset wrf --variables cape

# Download WRF variables but skip SST and sea ice (sourcing them elsewhere)
era5_dl parameters.toml --preset wrf --skip-vars sstk,ci

# List all available files without downloading
era5_dl parameters.toml --preset all --list-only

Usage

Basic invocation with a parameters file:

era5_dl parameters.toml --preset wrf

Override specific TOML values via CLI options:

era5_dl parameters.toml --preset wrf -s 2024-01-01 -e 2024-12-31 -n 4
era5_dl parameters.toml --preset wrf --min-lon 160 --max-lon 180 --no-check-target

List matching files without downloading (does not require [remote] config):

era5_dl parameters.toml --preset wrf --list-only

See all options:

era5_dl --help

CLI Options

All options are optional and override the corresponding value in the parameters TOML file.

Option Short Type Overrides
--n-tasks -n int n_tasks
--work-dir -w path work_dir
--check-target / --no-check-target bool check_target
--start-date -s text dates.start_date
--end-date -e text dates.end_date
--min-lon float bounds.min_lon
--max-lon float bounds.max_lon
--min-lat float bounds.min_lat
--max-lat float bounds.max_lat
--preset -p text preset
--variables -v text variables
--skip-vars text skip_vars
--list-only -l flag Query and list files only, skip download

Exit status and messages

Since 0.2.2, era5_dl exits 1 if any file failed at any step, or if listing a source directory failed, so a caller can trust exit 0 to mean every selected file reached the target. Each step (download, clip, upload) is judged by its tool's return code, not by whether the tool wrote to stderr: rclone writes to stderr on success too, for example ERROR : Attempt 1/3 failed before a retry succeeds. Before 0.2.2 such a recovered transfer counted as a failure, the good file was deleted, and the run still exited 0.

  • NOTICE <step> <key>: <message> (stdout): the step succeeded but its tool wrote to stderr. The file is kept.
  • FAILED <step> <key> rc=<code>: <message> (stderr): the file did not reach the target.
  • -- <n> of <m> files FAILED (stderr), then exit 1.

--check-target (the default) lists the target with rclone lsf and downloads every selected file it does not hold. A target that cannot be listed is an error (exit 1). Before 0.2.2 this used rclone check --missing-on-dst, which exits 1 for an error as for a difference, and its empty output on an error was read as the key '', a copy of the whole source root.

Metadata

Release files for era5-s3-dl 0.2.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for era5-s3-dl 0.2.2
File Size Uploaded Provenance Actions
era5_s3_dl-0.2.2.tar.gz 18.6 kB Without provenance Details for era5_s3_dl-0.2.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for era5-s3-dl 0.2.2
File Interpreter ABI Platform Actions
era5_s3_dl-0.2.2-py3-none-any.whl Python 3 none any Details for era5_s3_dl-0.2.2-py3-none-any.whl

Total release size: 36.3 kB

Release files / era5_s3_dl-0.2.2.tar.gz

Download URL era5_s3_dl-0.2.2.tar.gz
Size 18.6 kB
Tags Source
SHA-256 checksum
How to use checksums
517c68f6db1a6ea93abe6a4fa5a840bb7d60fd0cc4d2437aa0a4ba6cc7f8879c
BLAKE2b-256 checksum
How to use checksums
02cc2b3dc1ce5d159fae551268b64817adb44d3c7bb90ae1ef964e136d33f802
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.8.7

Release files / era5_s3_dl-0.2.2-py3-none-any.whl

Download URL era5_s3_dl-0.2.2-py3-none-any.whl
Size 17.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
034fec5cbc4b6a0379fdbd00bc95563113b110ee8c995a4b3e52781d44e237a8
BLAKE2b-256 checksum
How to use checksums
e093187c861d8b891f87be08e15d6961c0a8ac9658f608cf4e0004f07f35001b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.8.7

Security for era5-s3-dl 0.2.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page