mlcast
⚠️ This package is under active development. The API and functionality are subject to change until the v1.0.0 release.
The MLCast Community is a collaborative effort bringing together meteorological services, research institutions, and academia across Europe to develop a unified Python package for AI-based nowcasting. This is an initiative of the E-AI WG6 (Nowcasting) of EUMETNET.
This repo contains the mlcast package for machine learning-based weather nowcasting.
Installation
As mlcast is in rapid development — the recommended path is to clone locally,
rather than installing a pinned release from PyPI.
Local development: clone and install locally with uv
Fork the repository on GitHub first, then clone your fork. This lets you track upstream changes while keeping your own modifications on a separate branch:
git clone https://github.com/<your-github-username>/mlcast
cd mlcast
# Install uv if not already installed
curl -LsSf https://astral.sh/uv/install.sh | sh
# And install dependencies, picking the PyTorch build for your hardware.
# CPU only
uv sync --extra cpu
# GPU — CUDA 12.6
uv sync --extra gpu-cu126
# GPU — CUDA 12.8
uv sync --extra gpu-cu128
# GPU — CUDA 13.0
uv sync --extra gpu-cu130
PyTorch is only installed through one of these extras, so pick one: plain uv sync leaves it out. After this
one-time sync, uv run mlcast ... keeps the PyTorch build you chose.
Next you can jump to using mlcast or, if you intend to modify the code, setup the development toolchain as described below:
# Install dev dependencies, together with the PyTorch extra you picked above
# (`uv sync` removes extras that aren't listed, including PyTorch)
uv sync --extra cpu --extra dev
# Install the pre-commit git hook (runs checks automatically on every commit)
uv run pre-commit install
PyPI release
Tagged releases are published to PyPI and can be installed with pip. PyTorch comes through an extra, so include one:
pip install "mlcast[cpu]"
With pip, all of the extras (cpu, gpu-cu126, gpu-cu128, gpu-cu130) install the default PyTorch wheels from
PyPI. Install PyTorch first from the PyTorch index if you need a
specific build.
For active development or access to unreleased changes, clone the repository and install locally with uv as described above.
Usage
mlcast exposes two interfaces for training: a command-line interface (CLI) for interactive and scripted use, and a Python API for programmatic control. Both are built on Fiddle — a configuration library that lets you build a full experiment graph, override any parameter, and reproduce runs exactly from a saved YAML file.
Configuration model
Training in mlcast is currently built around a single base configuration
function, training_experiment, which defines the
default ConvGRU ensemble nowcasting setup: dataset, data module, network,
Lightning module, and trainer. Rather than writing a new config from scratch,
the intended workflow is to start from this base and apply targeted
modifications:
set:overrides — change a single scalar parameter (e.g. batch size, learning rate, number of epochs)- fiddlers — apply a named mutator function that keeps multiple related parameters in sync (e.g. switching the dataset class, toggling masking, changing the logger)
- direct graph edits (Python API only) — replace a sub-object entirely, for example swapping in a different network architecture
Any combination of these can be layered on top of the base config, and the fully resolved config is always saved to YAML alongside the training logs so runs can be reproduced exactly.
The diagram below shows the full default config graph as built by
training_experiment:
Command-line interface
Install the package and run:
mlcast train
This trains with the built-in training_experiment defaults. All parameters
are controlled via --config flags:
| Prefix | Purpose | Example |
|---|---|---|
| (none) | Use the built-in default config | mlcast train |
set: |
Override a single parameter | --config set:data.batch_size=32 |
fiddler: |
Apply a semantic mutator (multi-param change) | --config fiddler:use_random_sampler |
config: |
Switch to a different @auto_config function |
--config=config:my_experiment |
path/to/config.yaml |
Load a previously saved config | --config saved.yaml |
Multiple --config flags are applied in order and can be combined freely.
Examples:
# Override dataset path and batch size
mlcast train \
--config set:data.dataset_factory.zarr_path=/data/radar.zarr \
--config set:data.batch_size=32
# Switch to random sampler and log to MLflow
mlcast train \
--config fiddler:use_random_sampler \
--config fiddler:use_mlflow_logger
# Resume from a saved config with an epoch override
mlcast train \
--config logs/mlcast/version_0/config.yaml \
--config set:trainer.max_epochs=50
# Inspect the fully resolved config without starting training
mlcast train --config fiddler:use_random_sampler --print_config_and_exit
Run mlcast train --help for a full list of examples and available fiddlers.
Beyond training, the CLI provides two data-prep subcommands —
mlcast build-sampling-index and mlcast validate-sampling-index — for building
and checking the sampling index that training reads. See
Preparing training data.
Python API
The Python API gives you full programmatic control over the config graph before anything is instantiated.
Run the default experiment with tweaks:
import fiddle as fdl
from mlcast.config import training_experiment, train_from_config
from mlcast.config.fiddlers import use_random_sampler
cfg = training_experiment.as_buildable() # returns a fdl.Config graph — see src/mlcast/config/base.py
# Apply a fiddler to switch the dataset sampler
use_random_sampler(cfg)
# Override individual parameters directly on the config graph
cfg.data.batch_size = 32
cfg.trainer.max_epochs = 50
# Validates cross-parameter contracts, builds all objects, persists config
# YAML to the active logger, then calls trainer.fit() + trainer.test()
train_from_config(cfg)
Custom network architecture:
You can swap in any architecture by replacing cfg.pl_module.network with a
fdl.Config node. The network must implement the nowcasting forward
interface — see Custom network interface below.
As an example, here is how to wrap an
mfai HalfUNet (a plain single-step
U-Net) to satisfy the interface. The wrapper channel-stacks the past frames
and runs the U-Net autoregressively for each requested forecast step:
Note —
input_stepsequalsdataset_factory.input_steps(6 by default) and is directly readable from the config graph before building.
import einops
import fiddle as fdl
import torch
import torch.nn as nn
from jaxtyping import Float
from mfai.torch.models import HalfUNet
from mlcast.config import training_experiment, train_from_config
from mlcast.config.fiddlers import use_random_sampler
# Minimal adapter: channel-stack past frames → HalfUNet → one step at a time.
# NowcastLightningModule calls network(x, steps=N, ensemble_size=M), so any
# custom network must accept those keyword arguments.
class HalfUNetNowcaster(nn.Module):
def __init__(self, input_steps: int = 6, num_vars: int = 1):
super().__init__()
self.input_steps = input_steps
self.num_vars = num_vars
self.unet = HalfUNet(
input_shape=(256, 256),
in_channels=input_steps * num_vars,
out_channels=num_vars,
settings=fdl.Config(HalfUNet.settings_kls),
)
@property
def input_channels(self) -> int:
# Externally, the HalfUNetNowcaster respects the required input shape structure
# (batch, input_steps, num_vars, H, W), even though the internal U-Net is channel-stacked.
# Adding this property allows the config consistency checks to verify that
# the dataset and model agree on the expected number of input channels.
return self.num_vars
def forward(
self,
x: Float[torch.Tensor, "batch input_steps in_channels H W"],
steps: int,
ensemble_size: int = 1,
) -> Float[torch.Tensor, "batch steps out_channels H W"]:
# channel-stack all input frames: (b, t, c, h, w) -> (b, t*c, h, w)
x_flat = einops.rearrange(x, "b t c h w -> b (t c) h w")
preds = []
for _ in range(steps):
y = self.unet(x_flat) # [B, num_vars, H, W]
preds.append(y.unsqueeze(1))
# slide window: drop the oldest timestep (first num_vars channels),
# append the latest prediction as the newest timestep
x_flat = torch.cat([x_flat[:, self.num_vars:], y], dim=1)
return torch.cat(preds, dim=1)
cfg = training_experiment.as_buildable()
use_random_sampler(cfg)
cfg.pl_module.network = fdl.Config(
HalfUNetNowcaster,
input_steps=cfg.data.dataset_factory.input_steps,
num_vars=len(cfg.data.dataset_factory.standard_names),
)
train_from_config(cfg)
For lower-level control you can call the steps of train_from_config individually:
import fiddle as fdl
from mlcast.config.consistency_checks import validate_config
validate_config(cfg) # raises ValueError on any contract violation
experiment = fdl.build(cfg) # instantiates all objects
experiment.run() # trainer.fit() + trainer.test()
Available fiddlers
| Fiddler | Arguments | What it does |
|---|---|---|
use_mlflow_logger |
(none) | Replaces the default TensorBoardLogger with MLFlowLogger and appends LogSystemInfoCallback; respects the MLFLOW_TRACKING_URI environment variable |
set_variables |
standard_names |
Sets the list of input variables on the dataset and updates network.input_channels to match |
toggle_masking |
enabled |
Toggles masked-loss mode by setting both dataset_factory.return_mask and pl_module.masked_loss to the same value |
use_anon_s3_dataset |
zarr_path, endpoint_url |
Points the dataset at an anonymous S3 object store; sets zarr_path and the required storage_options together |
use_ratio_splits |
train, val |
Sets fraction-based train/val/test time splits on the data module (test = 1 − train − val) |
use_random_sampler |
(none) | Switches the dataset factory to the on-the-fly random sampler (useful during development when no precomputed sampling index is available) |
Preparing training data
Training reads a sampling index: a parquet file listing every candidate
datacube — a fixed-size time_depth × width × height crop of a source radar
Zarr — together with its sample stats (nan_count, sum, mean,
frac_wet), while the sampling parameters used to build it are stored in
the file's metadata. The training dataset
SourceDataIndexedDataset iterates
this index (it is the dataset factory's index_path), and a candidate
selector picks the training subset at dataset init. The index builder, its
contract, and the selectors all live in
mlcast.data.source_data.sampling;
two CLI subcommands build and check the index.
mlcast build-sampling-index — scan a Zarr dataset → sampling index
Slides a window over the (time, x, y) grid and, for every candidate datacube,
computes its sample stats in O(1) per window via a cumulative-sum trick.
Windows are filtered by a maximum-NaN budget, a spatial/temporal stride, and
time-continuity (no frame gaps); the survivors are streamed to a single
zstd-compressed parquet whose footer carries every sampling parameter as
metadata (the contract in
sampling_index_spec),
so downstream commands never have to parse the filename.
# A year of radar → 24-frame 256×256 datacubes, stride 3×16×16
mlcast build-sampling-index /data/radar.zarr \
--start-date 2020-01-01 --end-date 2020-12-31 \
--time-depth 24 --width 256 --height 256 \
--step-t 3 --step-x 16 --step-y 16 \
--max-nan 10000 \
-o sampling_index_2020.parquet
Common flags (mlcast build-sampling-index -h lists them all):
| Flag | Default | Purpose |
|---|---|---|
--time-depth / --width / --height |
24 / 256 / 256 | Datacube shape (T, X, Y) |
--step-t / --step-x / --step-y |
3 / 16 / 16 | Stride between candidate window starts |
--max-nan |
10000 | Drop any datacube with more NaNs than this |
--wet-threshold |
auto | Wet-pixel threshold; auto = 0.1 mm/h (rain rate) or 7 dBZ (reflectivity) |
--device |
auto | Compute backend: auto / cpu (bottleneck) / cuda |
--workers |
8 | CPU worker processes, or GPU chunk-reader threads |
--data-var / --time-var |
RR / time | Names of the Zarr data and time variables |
-o / --output |
auto | Output path; auto-named from the parameters if omitted |
mlcast validate-sampling-index — check an index against the contract
Checks a sampling index's column schema, sampling parameters, and (unless
--no-data-checks) its per-row value invariants, then prints the parameters,
the table structure, and a preview of the first 10 rows.
mlcast validate-sampling-index sampling_index_2020.parquet
# Footer only — schema + sampling parameters, skip the per-row checks
mlcast validate-sampling-index sampling_index_2020.parquet --no-data-checks
Candidate selectors
At training time SourceDataDataModule
applies a CandidateSelector to the index once, at dataset init, turning
the full candidate pool into the training set via a per-row keep/discard draw —
so the dataset length is fixed and known up front, and the same set is reused
every epoch (candidates are not re-drawn). Selectors are pluggable through a
registry in selection:
| Selector | Parameters | What it does |
|---|---|---|
UniformSelector |
keep_fraction |
Keep each candidate with a fixed probability, independent of its stats |
ImportanceSelector |
column, q_min, scale, mean_weight |
Keep each candidate with probability rising with one of its sample stats (mean by default) — oversampling high-rainfall datacubes without duplication |
The default config sets train_selector=ImportanceSelector() and
eval_selector=UniformSelector(keep_fraction=0.1), so importance sampling
reshapes only training while validation and test stay representative. Add a
scheme by subclassing CandidateSelector, decorating it with
@register_selector("name"), and selecting it from a config via
get_selector("name", ...).
Project Structure
mlcast/
├── src/mlcast/
│ ├── __main__.py # CLI entry point (train / build-sampling-index / validate-sampling-index)
│ ├── nowcasting_module.py # Generic Lightning module for nowcasting
│ ├── losses.py # CRPS, AFCRPS, MSE loss functions
│ ├── callbacks.py # Training callbacks
│ ├── visualization.py # TensorBoard image logging helpers
│ ├── config/
│ │ ├── base.py # Default training_experiment @auto_config
│ │ ├── fiddlers.py # Semantic config mutators
│ │ ├── consistency_checks.py # Cross-parameter validation
│ │ ├── loader.py # YAML config loader
│ │ └── orchestrator.py # train_from_config, config persistence
│ ├── data/
│ │ ├── source_data_datamodule.py # Lightning DataModule
│ │ ├── source_data_datasets.py # Zarr-backed PyTorch datasets (read the sampling index)
│ │ ├── normalization.py # Normalisation registry
│ │ └── source_data/sampling/ # Data-prep: sampling index builder + candidate selectors
│ │ ├── commands/ # build-sampling-index / validate-sampling-index CLI
│ │ ├── selection.py # Candidate selector registry (uniform, importance)
│ │ ├── sampling_index_spec.py # Sampling index schema + validation (the contract)
│ │ └── units.py # Data-kind + wet-threshold detection
│ └── models/
│ └── convgru.py # ConvGRU encoder-decoder
├── tests/
├── pyproject.toml
└── README.md
Implemented architectures
ConvGruModel
ConvGruModel (in src/mlcast/models/convgru.py) is an encoder-decoder
architecture. It is not autoregressive at forecast time: rather than
generating each forecast frame from the previous predicted frame, the decoder
performs a temporal roll-out entirely in latent space — the ConvGRU at
each spatial scale unrolls over forecast_steps steps driven by noise or
zeros, with its hidden state initialised from the encoder. Forecast frames
are only materialised at the end, by upsampling the final decoder hidden
states back to the original spatial resolution.
Encoding — a stack of EncoderBlock layers unrolls a ConvGRU
sequentially over the input_steps real observed frames. Each block halves
the spatial resolution via PixelUnshuffle(2). The last hidden state of
each block is retained.
Decoding — a stack of DecoderBlock layers performs a latent-space
roll-out at each spatial scale. Each decoder block's ConvGRU is initialised
with the final hidden state from the corresponding encoder block, then unrolls
over forecast_steps steps with noise or zeros as input — so the forecast
sequence emerges from the evolution of hidden states across multiple spatial
scales, never from feeding predictions back as inputs. Spatial resolution is
doubled at each block via PixelShuffle(2).
Ensemble — when ensemble_size > 1 the decoder is run ensemble_size
times, each time with freshly sampled Gaussian noise. The results are
concatenated along the channel dimension.
Deterministic variant (diagram source):
Stochastic / ensemble variant (diagram source):
Custom network interface
Any network architecture can be used by replacing cfg.pl_module.network
with a fdl.Config node pointing at your class. The only requirement is
that forward accepts the following signature:
# from jaxtyping import Float
# import torch
def forward(
self,
x: Float[torch.Tensor, "batch input_steps in_channels H W"],
steps: int, # number of forecast steps to produce
ensemble_size: int, # number of stochastic ensemble members
) -> Float[torch.Tensor, "batch steps out_channels H W"]:
...
If your network uses a different parameter name for the input channel count
than input_channels (the default assumed by ConvGruModel and the
set_variables fiddler), set it explicitly on the config node.
Contributing
Please feel free to raise issues or PRs if you have any suggestions or questions.
Links to presentations for discussion about the API
License
This project is dual-licensed under either:
- Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
- BSD 3-Clause License (LICENSE-BSD or https://opensource.org/licenses/BSD-3-Clause)
at your option.
See LICENSE for more details.
Metadata
Release files for mlcast 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| mlcast-0.2.0.tar.gz | 710.0 kB | With provenance | Details for mlcast-0.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| mlcast-0.2.0-py3-none-any.whl | Python 3 | none | any | Details for mlcast-0.2.0-py3-none-any.whl |
Total release size: 791.1 kB
Release files / mlcast-0.2.0.tar.gz
| Download URL | mlcast-0.2.0.tar.gz |
|---|---|
| Size | 710.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7b9e6cfd4750d3b050ddbd1a62437a461858706cb879da71058d3bebb5c26936
|
|
BLAKE2b-256 checksum How to use checksums |
207f2955d7c25b2364eb40a5e021f725a496b13f79914c7dc0c3797011ef10e4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 4, 2026.
Transparency logRelease files / mlcast-0.2.0-py3-none-any.whl
| Download URL | mlcast-0.2.0-py3-none-any.whl |
|---|---|
| Size | 81.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9e2357e9bb8585908469a7537b25d61eca8d9e1acc5c1e2dd5fc640c1fe2a782
|
|
BLAKE2b-256 checksum How to use checksums |
d7b49897d74c8055846d84a220d7c1f00b5d6d523122848697460720b2fbfbaf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 4, 2026.
Transparency logSecurity for mlcast 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for mlcast-0.2.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 4, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.