Skip to main content

Northflow

climval

climval is a Python library that validates and benchmarks climate model output against reference datasets.

Research page · PyPI · Notebooks · Licence

PyPI CI Licence: Apache 2.0

Why climval

Every lab compares climate models with its own scripts for RMSE, bias and Taylor diagrams, and the results rarely reproduce across groups. climval gives that comparison a typed, composable Python interface that runs the same way on a laptop, in a CI pipeline and in a long research workflow. It works with CMIP6, ERA5, CORDEX and custom simulation data.

Features

  • Metric library: RMSE, MAE, mean bias, normalised RMSE, Pearson r, Spearman r, Taylor skill score and percentile bias (P5, P95)
  • Typed data models: CF-convention variable definitions, spatial and temporal domain objects, and structured ValidationResult schemas
  • Presets: ERA5, ERA5-Land, CMIP6-MPI, CMIP6-HadGEM and CORDEX-EUR
  • NetCDF, CSV and Zarr: read through xarray when real files are provided
  • Synthetic fallback: structural validation and CI without data files. Runs on synthetic data emit a UserWarning and are flagged in the report; those scores are not a validation
  • Export: HTML report, JSON and Markdown
  • CLI: run comparisons from the terminal
  • Extensible: add your own metrics, variables and loaders

Quickstart

from datetime import datetime
import numpy as np
from climval import BenchmarkSuite
from climval.metrics import RMSE, MAE, MeanBias, PearsonCorrelation, TaylorSkillScore
from climval.models.schema import (
    ClimateModel, ModelType, SpatialDomain,
    TemporalDomain, SpatialResolution, TemporalResolution,
    ClimateVariable
)

# Define your reference dataset (e.g. ERA5)
era5 = ClimateModel(
    name="ERA5",
    model_type=ModelType.REANALYSIS,
    spatial_domain=SpatialDomain(
        lat_min=39.0, lat_max=40.0,
        lon_min=-0.5, lon_max=0.5,
    ),
    temporal_domain=TemporalDomain(
        start=datetime(2025, 6, 1),
        end=datetime(2025, 6, 30),
    ),
    spatial_resolution=SpatialResolution.LOW,
    temporal_resolution=TemporalResolution.HOURLY,
    variables=[
        ClimateVariable(name="tas", long_name="Near-Surface Air Temperature",
                        units="K", standard_name="air_temperature")
    ],
    metadata={"source": "Copernicus Climate Data Store (CDS)"},
)

# Define your candidate dataset (e.g. Sentinel-3 SLSTR LST)
sentinel3 = ClimateModel(
    name="Sentinel-3 SLSTR LST",
    model_type=ModelType.OBSERVATION,
    spatial_domain=SpatialDomain(
        lat_min=39.0, lat_max=40.0,
        lon_min=-0.5, lon_max=0.5,
    ),
    temporal_domain=TemporalDomain(
        start=datetime(2025, 6, 1),
        end=datetime(2025, 6, 30),
    ),
    spatial_resolution=SpatialResolution.HIGH,
    temporal_resolution=TemporalResolution.DAILY,
    variables=[
        ClimateVariable(name="tas", long_name="Land Surface Temperature",
                        units="K", standard_name="surface_temperature")
    ],
    metadata={"source": "EOPF Sentinel Zarr Samples Service (EODC)"},
)

# Run benchmark
suite = BenchmarkSuite(name="S3-LST-vs-ERA5")
suite.register(era5, role="reference")
suite.register(sentinel3)

report = suite.run(variables=["tas"], seed=42)
report.summary()
report.export("validation_report.html")

Output:

════════════════════════════════════════════════════════════════════════
  CLIMVAL: EUROPE-2000-2020
  Reference : ERA5
  Generated : 2025-01-15 14:32 UTC
════════════════════════════════════════════════════════════════════════

  ▸ Candidate: MPI-ESM1-2-HR
  ──────────────────────────────────────────────────────────────────────
  Variable   Metric                              Value  Units
  ──────────────────────────────────────────────────────────────────────
  tas        rmse                               1.2930  K
  tas        mae                                1.0744  K
  tas        mean_bias                         -0.9721  K
  tas        nrmse                              0.0845  dimensionless
  tas        pearson_r                          0.9985  dimensionless
  tas        taylor_skill_score                 0.9969  dimensionless
  tas        percentile_bias_p95               -0.0473  %
  tas        percentile_bias_p5                -0.6058  %
  ...

Installation

Core (no NetCDF dependencies):

pip install climval

With NetCDF/xarray support:

pip install "climval[netcdf]"

With visualization:

pip install "climval[viz]"

Development:

git clone https://github.com/northflowlabs/climval
cd climval
pip install -e ".[dev]"
pytest

CLI

# Run a validation
climval run \
  --reference era5 \
  --candidates cmip6-mpi cordex-eur \
  --variables tas pr \
  --output results/report.html \
  --name "Europe-2000-2020"

# List available metrics
climval list-metrics

# List model presets
climval list-presets

# Inspect a preset
climval info --model era5

Metrics

Metric Name Direction Reference
Root Mean Square Error rmse ↓ lower better
Mean Absolute Error mae ↓ lower better
Mean Bias mean_bias ↓ lower better
Normalized RMSE nrmse ↓ lower better
Pearson Correlation pearson_r ↑ higher better
Spearman Correlation spearman_r ↑ higher better
Taylor Skill Score taylor_skill_score ↑ higher better Taylor (2001)
Percentile Bias (P95) percentile_bias_p95 ↓ lower better Gleckler et al. (2008)
Percentile Bias (P5) percentile_bias_p5 ↓ lower better Gleckler et al. (2008)

Custom metrics

from climval.metrics import BaseMetric
import numpy as np

class KGE(BaseMetric):
    """Kling-Gupta Efficiency (Gupta et al., 2009)."""
    name = "kge"
    higher_is_better = True

    def compute(self, reference, candidate, weights=None):
        r     = np.corrcoef(reference, candidate)[0, 1]
        alpha = np.std(candidate) / np.std(reference)
        beta  = np.mean(candidate) / np.mean(reference)
        return float(1 - np.sqrt((r - 1)**2 + (alpha - 1)**2 + (beta - 1)**2))

suite.add_metric(KGE())

Model Presets

Preset Type Spatial res. Temporal res. Description
era5 Reanalysis ~31 km Hourly ECMWF ERA5 Global Reanalysis
era5-land Reanalysis ~9 km Hourly ERA5-Land High-Resolution
cmip6-mpi GCM ~100 km Monthly MPI-ESM1-2-HR (Max Planck)
cmip6-hadgem GCM ~100 km Monthly HadGEM3-GC31-LL (Met Office)
cordex-eur RCM ~12 km Daily EURO-CORDEX 0.11°

Custom models load from any NetCDF4, CSV, or Zarr source:

from climval import load_model
from climval.models import ModelType

model = load_model(
    name="MyModel-v2",
    path="data/simulation.nc",
    variables=["tas", "pr", "psl"],
    lat_range=(50.0, 60.0),
    lon_range=(10.0, 25.0),
    model_type=ModelType.CUSTOM,
)

Standard Variables

Follows CF Conventions v1.10. Built-in definitions:

CF Name Long Name Units
tas Near-Surface Air Temperature K
pr Precipitation kg m⁻² s⁻¹
psl Sea Level Pressure Pa
ua Eastward Wind m s⁻¹
va Northward Wind m s⁻¹
hurs Near-Surface Relative Humidity %

Export Formats

Format Extension Use case
JSON .json Machine-readable, pipeline integration
HTML .html Self-contained report, sharing
Markdown .md GitHub/GitLab, documentation

Architecture

climval/
├── core/
│   ├── loader.py       # Model loader with preset registry
│   ├── suite.py        # BenchmarkSuite orchestrator
│   └── report.py       # BenchmarkReport + multi-format export
├── metrics/
│   └── stats.py        # All metric implementations + registry
├── models/
│   └── schema.py       # Typed data models (CF-convention)
└── cli.py              # climval CLI

Climate model evaluation has mature tools. climval works alongside them.

ESMValTool

ESMValTool is the community standard for Earth System Model evaluation, backed by ESA, CMIP and a consortium of European research institutions. If you are running full diagnostic campaigns against CMIP5/6 with pre-defined recipes, regridding pipelines, and multi-decadal observational datasets, ESMValTool is the right tool.

It is configured through YAML recipes and installed through conda, which suits a dedicated climate modelling team.

PCMDI Metrics Package (PMP)

PMP, developed at Lawrence Livermore National Laboratory, provides a systematic framework for computing standardized performance metrics against observational references. It is widely cited and takes work to deploy outside a national lab.

ClimateLearn

ClimateLearn is a Python library oriented toward machine learning pipelines for climate, providing standardized data loading and evaluation for ERA5 and CMIP6. Relevant if your workflow is ML-first.

Where climval fits

climval sits at a different layer:

ESMValTool PMP climval
Installation Complex, conda-heavy Moderate pip install climval
Configuration YAML recipes Config files Pure Python API
Learning curve Days to weeks Days Minutes
CI/CD integration Difficult Difficult Native
Custom metrics Recipe-based Limited class MyMetric(BaseMetric)
Programmatic use Partial Partial First-class
Scope Full diagnostic suite Performance metrics Composable validation

Use climval when you need:

  • A reproducible validation step inside a Python pipeline or CI workflow
  • A composable API to integrate climate metrics into larger systems
  • Rapid prototyping, teaching, or lightweight institutional reporting
  • Full programmatic control without YAML configuration or recipe management

Use ESMValTool when you need:

  • Pre-built diagnostic recipes from the CMIP community
  • Automated regridding and unit harmonization across large multi-model ensembles
  • Figures and diagnostics that conform to IPCC or CMIP reporting standards

climval is scoped to be the layer you can import in three lines.

References

  • Gleckler, P. J., Taylor, K. E., & Doutriaux, C. (2008). Performance metrics for climate models. Journal of Geophysical Research, 113, D06104. https://doi.org/10.1029/2007JD008972

  • Taylor, K. E. (2001). Summarizing multiple aspects of model performance in a single diagram. Journal of Geophysical Research, 106(D7), 7183-7192. https://doi.org/10.1029/2000JD900719

  • Gupta, H. V., Kling, H., Yilmaz, K. K., & Martinez, G. F. (2009). Decomposition of the mean squared error and NSE: Implications for improving hydrological modelling. Journal of Hydrology, 377(1-2), 80-91. https://doi.org/10.1016/j.jhydrol.2009.08.003

Contributing

Issues and pull requests are welcome. See CONTRIBUTING.md.

git clone https://github.com/northflowlabs/climval
cd climval
pip install -e ".[dev]"
pytest && ruff check . && mypy climval/

Licence

climval is licensed under the Apache License 2.0.

Copyright 2026 Northflow Technologies AS

Metadata

Release files for climval 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for climval 0.1.1
File Size Uploaded Provenance Actions
climval-0.1.1.tar.gz 30.5 kB With provenance Details for climval-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for climval 0.1.1
File Interpreter ABI Platform Actions
climval-0.1.1-py3-none-any.whl Python 3 none any Details for climval-0.1.1-py3-none-any.whl

Total release size: 56.9 kB

Release files / climval-0.1.1.tar.gz

Download URL climval-0.1.1.tar.gz
Size 30.5 kB
Tags Source
SHA-256 checksum
How to use checksums
faf2d1361d6bd4ac52ce112d9e7a239ac681c34bc862e2705c2fafdab074b5ce
BLAKE2b-256 checksum
How to use checksums
a5785f4d4b0d344dad23b743c5d669b403eceb8e00b83c2c97011fda579b1c5f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.12.8

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 29, 2026.

Transparency log

Release files / climval-0.1.1-py3-none-any.whl

Download URL climval-0.1.1-py3-none-any.whl
Size 26.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
c42e1f6ab0501c0cfb2fcdc376b8a3622518644fda0f224fd1f407e0a1c799d2
BLAKE2b-256 checksum
How to use checksums
521b01be611fcaae5cde5d7715666a7eccece503a66a035004109bfefb1eb80a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.12.8

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 29, 2026.

Transparency log

Security for climval 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for climval-0.1.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 29, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page