cdsswarm

Concurrent CDS API downloader with an interactive Textual TUI and script mode.

Submit multiple CDS API requests and download them in parallel with a configurable number of workers. Monitor progress through an interactive terminal UI with an htop-style worker table, or run headless in script mode for CI/cron jobs.

Feedback welcome! This project is under active development. If you have suggestions, feature requests, or run into any issues, please open an issue on GitHub or send an email to b.giebl@protonmail.com.

Performance

How much faster is parallel downloading? Benchmark with 4 monthly ERA5 files (2m temperature, Germany bounding box):

Each CDS request spends most of its time waiting in the server queue and being processed — parallel workers overlap that wait. For large files, download parallelism helps too. Results vary with server load and time of day; larger batches benefit more from higher worker counts.

Installation

pip install cdsswarm

Or via conda-forge:

conda install -c conda-forge cdsswarm

For YAML request file support:

pip install "cdsswarm[yaml]"

For development (tests, pre-commit):

git clone https://github.com/bgiebl/cdsswarm.git
cd cdsswarm
pip install -e ".[dev]"

Prerequisites

A valid CDS API configuration file at ~/.cdsapirc:

url: https://cds.climate.copernicus.eu/api
key: <your-personal-access-token>

See the CDS API documentation for setup instructions.

Quick Start

Command Line

Create a request file requests.json:

[
  {
    "dataset": "reanalysis-era5-single-levels",
    "request": {
      "product_type": ["reanalysis"],
      "variable": ["2m_temperature"],
      "year": ["2024"],
      "month": ["01"],
      "day": ["01", "02", "03"],
      "time": ["12:00"],
      "data_format": "grib"
    },
    "target": "temperature_jan.grib"
  },
  {
    "dataset": "reanalysis-era5-single-levels",
    "request": {
      "product_type": ["reanalysis"],
      "variable": ["total_precipitation"],
      "year": ["2024"],
      "month": ["01"],
      "day": ["01", "02", "03"],
      "time": ["12:00"],
      "data_format": "grib"
    },
    "target": "precipitation_jan.grib"
  }
]

Run with 4 workers:

cdsswarm requests.json --workers 4

Python API

import cdsswarm

tasks = [
    cdsswarm.Task(
        dataset="reanalysis-era5-single-levels",
        request={
            "product_type": ["reanalysis"],
            "variable": ["2m_temperature"],
            "year": ["2024"],
            "month": ["01"],
            "day": ["01", "02", "03"],
            "time": ["12:00"],
            "data_format": "grib",
        },
        target="temperature_jan.grib",
    ),
    cdsswarm.Task(
        dataset="reanalysis-era5-single-levels",
        request={
            "product_type": ["reanalysis"],
            "variable": ["total_precipitation"],
            "year": ["2024"],
            "month": ["01"],
            "day": ["01", "02", "03"],
            "time": ["12:00"],
            "data_format": "grib",
        },
        target="precipitation_jan.grib",
    ),
]

results = cdsswarm.download(tasks, num_workers=4)

for r in results:
    if r.success:
        print(f"Downloaded {r.task.target}")
    else:
        print(f"Failed {r.task.target}: {r.error}")

CLI Reference

usage: cdsswarm [-h] [--version] [-w WORKERS] [-m {interactive,script,auto}]
                [--no-skip] [--resume | --no-resume] [--reuse | --no-reuse]
                [--max-retries MAX_RETRIES] [--output-dir OUTPUT_DIR]
                [--dry-run] [--log FILE] [--summary FILE] [--post-hook CMD]
                requests_file

In auto mode, the TUI is used when stdout is a TTY; otherwise it falls back to script mode.

Post-download hooks

The --post-hook option runs a shell command after each file is successfully downloaded. Use {file} and {dataset} as placeholders:

# Compress each file after download
cdsswarm requests.json --post-hook "gzip {file}"

# Convert GRIB to NetCDF with CDO
cdsswarm requests.json --post-hook "cdo -f nc copy {file} {file}.nc"

# Upload to S3
cdsswarm requests.json --post-hook "aws s3 cp {file} s3://my-bucket/cds/"

Hook failures produce a warning but do not mark the download as failed — the file is already on disk.

Request generation

The generate subcommand expands a template file into a full request file using Cartesian product expansion:

cdsswarm generate template.json -o requests.json
cdsswarm generate template.json --dry-run          # preview without writing

The template file must contain a single JSON object (not a list). If you pass a single-element list [{...}], it will be auto-unwrapped with a warning.

A template looks like a single request with a split_by field that lists which dimensions to expand:

{
  "dataset": "reanalysis-era5-single-levels",
  "request": {
    "product_type": ["reanalysis"],
    "variable": ["2m_temperature", "total_precipitation"],
    "year": ["2023", "2024"],
    "month": ["01", "02", "03"],
    "day": ["01", "02", "03"],
    "time": ["12:00"],
    "data_format": "grib"
  },
  "target": "output/{variable}_{year}_{month}.grib",
  "split_by": ["variable", "year", "month"]
}

This generates 2 × 2 × 3 = 12 separate tasks, one for each combination of variable, year, and month. Non-split fields (day, time, etc.) are shared across all tasks. The {placeholder} syntax in target fills in each combination's values.

Cancelling requests

The cancel subcommand cancels active CDS API requests on the server — useful for cleaning up after a crashed session or accidental submissions:

cdsswarm cancel                        # cancel all queued/running requests (new API only)
cdsswarm cancel abc-123 def-456        # cancel specific request IDs (both APIs)
cdsswarm cancel --yes                  # skip confirmation prompt

When no IDs are given, cdsswarm queries the CDS server for all active (accepted/running) requests and presents them for confirmation before cancelling. This requires the new CADS API (ecmwf-datastores). With the old cdsapi, you must provide specific request IDs.

Shell completion

Tab-completion for subcommands and flags is available for bash and zsh:

# Bash (add to ~/.bashrc)
eval "$(cdsswarm completion bash)"

# Zsh (add to ~/.zshrc)
eval "$(cdsswarm completion zsh)"

Session resume

cdsswarm automatically saves session state after each task completes. If a download session is interrupted (e.g. by Ctrl+C or a network failure), rerunning the same command picks up where it left off — completed tasks are skipped and failed/pending tasks are retried.

State files are stored in ~/.cache/cdsswarm/sessions/ (or $XDG_CACHE_HOME), keyed by request file path and output directory. Run logs are automatically saved to ~/.local/state/cdsswarm/logs/ (or $XDG_STATE_HOME).

cdsswarm requests.json -w 4             # interrupted — 50 of 100 tasks done
cdsswarm requests.json -w 4             # resumes from task 51
cdsswarm requests.json -w 4 --no-resume # force a fresh start

Configuration file

Settings can be stored in a .cdsswarm.toml file instead of passing CLI flags every time. CLI flags always take precedence.

Example .cdsswarm.toml:

workers = 8
max-retries = 5
mode = "script"
output-dir = "/data/downloads"
post-hook = "gzip {file}"

All CLI flags are supported as config keys (use hyphens, e.g. max-retries, post-hook, skip-existing).

Request File Format

List format

Each entry specifies its own dataset:

[
  {
    "dataset": "reanalysis-era5-single-levels",
    "request": { ... },
    "target": "output1.grib"
  },
  {
    "dataset": "reanalysis-era5-pressure-levels",
    "request": { ... },
    "target": "output2.grib"
  }
]

Compact format

Share a dataset across all requests:

{
  "dataset": "reanalysis-era5-single-levels",
  "requests": [
    { "request": { ... }, "target": "output1.grib" },
    { "request": { ... }, "target": "output2.grib" }
  ]
}

YAML

Both formats also work in YAML (requires pip install cdsswarm[yaml]):

dataset: reanalysis-era5-single-levels
requests:
  - request:
      product_type: [reanalysis]
      variable: [2m_temperature]
      year: ["2024"]
      month: ["01"]
      day: ["01"]
      time: ["12:00"]
      data_format: grib
    target: temperature.grib

The request dict accepts the same parameters as cdsapi.Client.retrieve().

Python API Reference

cdsswarm.Task(dataset, request, target)

A single CDS API download request.

cdsswarm.download(tasks, num_workers=4, skip_existing=True, reuse_jobs=True, max_retries=3, on_message=None, post_hook="")

Download multiple CDS API requests concurrently.

Returns a list[Result]. Returns an empty list if interrupted by KeyboardInterrupt.

cdsswarm.Result

cdsswarm.expand_template(template, split_by=None)

Expand a template dict into a list of Task objects via Cartesian product of the split_by dimensions.

cdsswarm.ConfigError

Raised for invalid configuration values in config files or CLI flags. Subclass of ValueError.

cdsswarm.RequestFileError

Raised for invalid or unrecognized request file formats. Subclass of ValueError.

TUI

The interactive TUI (terminal user interface) is built with Textual and is available via the CLI only. It shows an htop-style DataTable with one row per worker:

W  │W-State  │CDS Status │Filename          │Started  │Elapsed  │Size    │DL %   │Request ID
0  │ active  │ running    │era5_2024_01.grib │22:31:24 │2h30m05s │15.2 GB│48%    │af1e2306-28c3...
1  │ active  │ successful │era5_2024_02.nc   │22:31:25 │1h15m00s │8.1 GB │100% ✓ │b2f4a891-...

The layout has two tabs (Workers and Files), an info panel above the table, and a progress footer with an overall progress bar and ETA.

Key bindings:

Running Tests

pip install -e ".[dev]"
pytest -v

License

MIT