DEFAIR AI Demo - Cloud Masking CNN

Standalone demonstration of cloud masking using a lightweight CNN on MSG satellite data.

Overview

This package demonstrates an end-to-end ML pipeline for cloud detection:

1. Load MSG native format data (.nat files) using the MSG15NativeReaderPlugin (We will add more data formats and therefore collocation in the upcoming issues)
2. Generate cloud masks using IR temperature thresholds
3. Convert to Zarr format with ML-optimized chunking
4. Train a simple U-Net CNN for binary cloud/clear classification
5. Generate predictions and visualizations

Quick Start

Navigate to the package directory first:

cd packages/defair-ai-demo

Option 1: Using Makefile (Recommended)

# See all available commands
make help

# Install dependencies
make install

# Run end-to-end demo
make demo

# Run tests
make test

# Clean up generated files
make clean

Option 2: Using uv directly

# Install dependencies
uv sync --package defair-ai-demo

# Run end-to-end demo
uv run python scripts/run_end_to_end.py

# Run tests
uv run pytest tests/

The collocation AI use case notebook

msg_mtg_era5_collocation_ai_usecase.ipynb, in this directory, is the fuller demonstration: it reads MSG SEVIRI and MTG FCI through DEFAIR and the ERA5 GRIB with earthkit-data, reprojects all three onto one grid over Europe, writes an analysis-ready Zarr, and then trains and runs the cloud-mask model from this package on it.

From PyPI, install the package together with the plotting stack the notebook uses. On Linux, run the first line too, so pip takes the CPU build of PyTorch instead of the CUDA build, about 3 GB of downloads:

pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install defair-ai-demo "defair[notebooks]"

From a source checkout, the same environment comes from the repository root with:

uv sync --frozen --all-packages --group notebooks

--all-packages provides this package and PyTorch, --group notebooks provides cartopy for the plotting helpers in defair.notebook_helpers.

The notebook reads the public demo bucket of the DestinE Data Lake by default and needs no credentials, as described in Install DEFAIR. Two variables are specific to this notebook. COLLOCATION_SOURCE is s3 (the default) or local, and local reads staged data from test-data/collocation/ instead of the bucket. COLLOCATION_BASE is the folder that local input paths are relative to, and defaults to the notebook folder.

Configuration

All hyperparameters and settings are in config.yaml:

• Data settings: MSG file path, channels to use
• Training settings: Epochs, batch size, learning rate
• Model settings: Input/output channels
• Paths: Output directories and file locations

Important: Update data.msg_dir in config.yaml to point to your directory of MSG .nat files

To use a custom config:

make run --config=my_config.yaml
# Or with uv:
uv run python scripts/run_end_to_end.py --config my_config.yaml

Available Makefile Commands

Output

All outputs are saved to output/ (in package directory):

• msg_processed.zarr/ - Processed satellite data
• cloud_model.pt - Trained model weights
• predictions/ - Cloud mask visualizations

Architecture

• Model: Lightweight U-Net (3 encoder/decoder levels, ~500K parameters)
• Input: 3 channels (IR_108, VIS006, IR_039) @ 256x256 patches
• Output: Binary cloud mask [0, 1]
• Labels: Generated using IR temperature threshold (IR_108 < 270K = clouds)