DEFAIR

DEFAIR (Destination Earth Framework for preparing AI-Ready data) reads, transforms, aligns, and writes meteorological and Earth-observation data as Dask-backed xarray datasets. Its plugin system supports EUMETSAT product readers, local and S3-compatible storage, geospatial transformations, and NetCDF/Zarr output.

Installation

DEFAIR requires Python 3.10 or later. Install it from PyPI into a virtual environment, with the notebooks extra when you want to run the tutorials:

python -m venv .venv
source .venv/bin/activate
pip install "defair[notebooks]"

Install DEFAIR covers the requirements, the AI tutorial package, pinning a version and the object store credentials.

For development from a source checkout, install the uv-managed workspace:

make install

DEFAIR releases frequently. Pin the exact version in reproducible pipelines and check the release notes, which list the resolved ticket numbers, before upgrading.

First dataset

Dataset.from_source() accepts a local path, an S3 URI, a glob, or an explicit list of paths. The reader and source plugins are auto-detected when possible.

from defair_data import Dataset

dataset = Dataset.from_source(
    "/data/MSG3-SEVI-MSG15-product.nat",
    reader="msg15nat",
    channels=["IR_108"],
    calibration="brightness_temperature",
    use_channel_names=True,
)

print(dataset.data)
dataset.to_file("output/msg.zarr", writer="zarrv2")

Reader results are Dask-backed by default.

S3 source selection

For an s3:// URI, omitting source auto-selects source="s3". This is the s3fs/fsspec-backed plugin and is the normal choice for xarray and Dask workflows. source="s3boto" selects the direct boto3 plugin for source-level operations such as range-based file access, listing, and globbing.

dataset = Dataset.from_source(
    "s3://bucket/path/product.nc",
    source="s3",  # optional: this is the auto-detected S3 default
    source_kwargs={
        "endpoint_url": "https://s3.example.com",
        "aws_access_key_id": "...",
        "aws_secret_access_key": "...",
    },
    reader="mtg_fci_l1c_nc",
)

Both plugins pass the same read settings to xarray. s3boto changes how DEFAIR lists files and performs direct reads, but xarray readers may still use s3fs. See Data loading, S3, and streaming for the differences, credential options, and performance settings.

Large multi-file datasets

Normal multi-file loading builds one Dataset, while reading array data only when it is needed. For a large collection of files, use streaming=True. DEFAIR processes a fixed number of scene groups at a time and appends each group to Zarr:

cube = Dataset.from_source(
    "s3://bucket/archive/*.nc",
    reader="mtg_fci_l1c_nc",
    use_channel_names=True,
    streaming=True,
    stream_batch_size=4,
)

cube = cube.transform("content_filter", include_vars=["vis_06", "ir_105"])
cube.to_file("output/archive.zarr", writer="zarrv2")

A streaming plan does not provide .data. It records supported transformations and writes only to Zarr. Reload the Zarr to inspect the result. Use streaming=False when you need the complete dataset or an operation that combines all files.

Discovering capabilities and reader options

List registered plugins and inspect the selected reader's constructor and read() signatures. Options from both signatures can be passed through Dataset.from_source(). IDE completion on from_source() shows only its common parameters because reader-specific options travel through **kwargs:

from inspect import signature

from defair.plugin_manager import list_readers, list_sources, load_reader

print(list_sources())
print(list_readers())

reader = load_reader("msg15nat")
print(signature(type(reader)))  # calibration, chunks, use_channel_names, ...
print(signature(reader.read))   # channels, lazy_coordinates, ...

To resolve a reader short name to its product name and EUMETSAT collection ID, use describe_readers() (or defair readers on the command line):

from defair_data.readers import describe_readers, get_reader_info

print(describe_readers())
print(get_reader_info("mtg_fci_l2_amv"))  # EO:EUM:DAT:0676

The full table is in docs/reader-catalogue.md.

Reader arguments are validated against the reader's declared parameters, so an unrecognised option raises and lists the accepted arguments rather than being ignored. This applies to third-party reader plugins too: a **kwargs catch-all on read() does not widen what a reader accepts, so declare every supported option by name.

Documentation

• Getting started: what DEFAIR is, installation, a first product and a first workflow
• Data loading, S3, streaming, and reader options
• Workflow examples
• Dask cluster recommendations
• Performance benchmarks
• Tutorial notebooks, and the per-product coverage set under tests/notebooks/
• Collocation AI use case, the notebook that puts MSG, MTG and ERA5 on one grid and trains a cloud-mask model on it (the ERA5 leg is read with earthkit-data, which is how a format DEFAIR has no reader for reaches DEFAIR)
• Reader, writer, and transformation plugin development

The command-line entry point also lists the installed commands and options:

defair --help

Logging

Set logging with CLI flags or environment variables:

defair --log-level DEBUG version
defair --log-json version
defair --log-module-level "defair_data:DEBUG,defair_ops:WARNING" version

export DEFAIR_LOG_LEVEL=INFO
export DEFAIR_LOG_JSON=true

Python callers can configure logging directly:

from defair.logging import setup_logging

setup_logging(log_level="INFO", json_output=False)

Development

The uv workspace contains three publishable packages:

• defair (src/defair/): CLI and workflow orchestration
• defair-data (packages/defair-data/): data sources, readers, and writers
• defair-ops (packages/defair-ops/): transformations and Dask management

These are mutually dependent rather than strictly layered: each imports the other two, so each declares the other two, and installing any one of them pulls in the rest. The coupling exists because the shared kernel — the entry-point plugin manager, CFHistoryMixin, kwargs validation and logging — lives in defair rather than in a leaf package, so the lower-level packages call back up into it. A new cross-package import therefore has to be declared in that package's pyproject.toml, even though a workspace uv sync satisfies it either way.

Common development commands:

make install
make check
make test
make docs

Add a dependency to a workspace package with:

uv add --package <defair|defair-data|defair-ops> <dependency_name>

All Python source files require Apache 2.0 license headers. Run make headers-check to validate them or make headers-add to add missing headers.

Release maintenance

Build all workspace packages with:

uv build --all-packages

Version commands use uv's workspace support:

uv version --package <defair|defair-data|defair-ops> --bump <major|minor|patch|rc|stable>

For a patch to an older release, merge the fix to main, create a release branch from the target tag, cherry-pick the fix, update package versions, and tag the patched release.

Building the documentation

Build the local Material for MkDocs site with:

make docs

To generate the DEDL ReadTheDocs/Sphinx source tree, run:

uv run python scripts/generate_docs.py path/to/target/directory