stationbench 0.1.0

A benchmarking tool for weather forecasts based on weather stations

StationBench

StationBench is a Python library for benchmarking weather forecasts against weather station data. It provides tools to calculate metrics, visualize results, and compare different forecast models.

Features

• Pre-processed ground truth data from 10,000+ weather stations around the world included in the package
• Calculate RMSE and other metrics between forecasts and ground truth data
• Support for multiple weather variables (temperature, wind speed, solar radiation)
• Regional analysis capabilities (Europe, North America, Global, etc.)
• Integration with Weights & Biases for experiment tracking

Installation

Using poetry:

poetry install

Documentation

Full documentation is available in the docs/ directory:

• Setup - How to setup StationBench
• Tutorial - Basic usage of StationBench

Quick Start

Data Format Requirements

Forecast Data

• Must include dimensions: latitude, longitude, time
• Variables should include:
  • 10m_wind_speed (or custom name)
  • 2m_temperature (or custom name)

Ground Truth Data

Stationbench comes with ready-to-use weather stations from around the world. The benchmarking data is a subset of the Meteostat dataset. It contains weather data from 2018-2024 for 10m wind speed and 2m temperature. The data is provided by the following organizations:

• Deutscher Wetterdienst
• NOAA
• Government of Canada
• MET Norway
• European Data Portal
• Offene Daten Österreich

Source: Meteostat (CC BY-NC 4.0)

The benchmarking data can be accessed from https://opendata.jua.sh/stationbench/meteostat_benchmark.zarr.

Besides the provided benchmarking data, you can also use your own ground truth data. The ground truth data must be in zarr format and must include the following dimensions and coordinates:

• Must include dimensions: station_id, time
• Must include coordinates: latitude, longitude

Calculate Metrics

This script computes metrics (RMSE only for now) by comparing forecast data against ground truth data for specified time periods and regions. Output are RMSE benchmarks for different variables and lead times in the format of the ground truth data.

Options

• --forecast_loc: Location of the forecast data (required)
• --ground_truth_loc: Location of the ground truth data (defaults to https://opendata.jua.sh/stationbench/meteostat_benchmark.zarr)
• --start_date: Start date for benchmarking (required)
• --end_date: End date for benchmarking (required)
• --output_loc: Output path for benchmarks (required)
• --region: Region to benchmark (see regions.py for available regions)
• --name_10m_wind_speed: Name of 10m wind speed variable (optional)
• --name_2m_temperature: Name of 2m temperature variable (optional)

If variable name is not provided, no metrics will be computed for that variable.

Example usage

poetry run python stationbench/calculate_metrics.py \
    --forecast_loc forecast.zarr \
    --start_date 2023-01-01 --end_date 2023-12-31 --output_loc forecast-rmse.zarr \
    --region europe --name_10m_wind_speed "10si" --name_2m_temperature "2t"

Compare forecasts

After generating the metrics, you can use the compare_forecasts.py script to compute metrics, create visualizations, and log the results to Weights & Biases (W&B).

What it does

The compare_forecasts.py script:

1. Computes RMSE (Root Mean Square Error) and skill scores for different variables and lead time ranges.
2. Generates geographical scatter plots showing the spatial distribution of errors.
3. Creates line plots showing the temporal evolution of errors.
4. Logs all visualizations and metrics to a W&B run.

Options

• --evaluation_benchmarks_loc: Path to the evaluation benchmarks (required)
• --reference_benchmark_locs: Dictionary of reference benchmark locations, the first one is used for skill score (required)
• --run_name: W&B run name (required)
• --regions: Comma-separated list of regions, see regions.py for available regions (required)

Example

poetry run python stationbench/compare_forecasts.py \
    --evaluation_benchmarks_loc forecast-rmse.zarr \
    --reference_benchmark_locs '{"HRES": "hres-rmse.zarr"}' \
    --regions europe \
    --run_name wandb-run-name

Contributing

We welcome contributions! Please see our CONTRIBUTING.md for details.

License

This project is licensed under the MIT License - see the LICENSE file for details.