Skip to main content

MLCast Community intake catalog

data-availability-check linting Jupyter Book Badge

Hi! 👋

You are looking at the source data intake catalog for the MLCast community. This is a collection of datasets we have currated with the aim of making them available to build machine learning training datasets from.

The following diagram shows the intended data flow and how the intake catalog (this repository) fits into the overall architecture of the MLCast project.

source for this graphic

How to use this catalog

To use the catalog, you can either a) install the necessary python packages yourself and read the catalog directly from github or b) install the most recent tagged release of the mlcast_datasets python package from pypi.org and read the catalog included in that release. Reading the catalog from github is useful if you want to use the most recent version of the catalog, while installing the mlcast_datasets package is useful if you want to use a stable version of the catalog.

a) Reading the catalog directly from github

To read and open datasets in the catalog you will need to have the following packages installed:

pip install intake intake-xarray zarr jinja2

Or, you can installing the mlcast-datasets package directly from this repository, which will install all the necessary dependencies:

pip install git+https://github.com/mlcast-community/mlcast-datasets

The catalogue (and underlying data) can then be accessed directly from python:

import intake
cat = intake.open_catalog("https://raw.githubusercontent.com/mlcast-community/mlcast-datasets/main/src/mlcast_datasets/catalog/catalog.yml")

b) Installing the mlcast_datasets package

To install the most recent tagged release of the mlcast_datasets package, you can use pip:

pip install mlcast-datasets

and then read the catalog from the package:

import mlcast_datasets
cat = mlcast_datasets.open_catalog()

Using data within the catalog

Once you have opened the catalog, you can list the available sources with:

>> list(cat)
['precipitation']

>> list(cat.precipitation)
['radklim_hourly', 'radklim_5_minutes']

Then load up a dask-backed xarray.Dataset so that you have access to all the available variables and attributes in the dataset:

>> ds = cat.precipitation.radklim_5_minutes.to_dask()
>> ds
<xarray.Dataset> Size: 10TB
Dimensions:          (time: 2419200, y: 1100, x: 900)
Coordinates:
  * time             (time) datetime64[ns] 19MB 2001-01-01 ... 2023-12-31T23:...
  * y                (y) float64 9kB -4.758e+03 -4.757e+03 ... -3.659e+03
  * x                (x) float64 7kB -443.0 -442.0 -441.0 ... 454.0 455.0 456.0
    lat              (y, x) float64 8MB dask.array<chunksize=(1100, 900), meta=np.ndarray>
    lon              (y, x) float64 8MB dask.array<chunksize=(1100, 900), meta=np.ndarray>
Data variables:
    rainfall_amount  (time, y, x) float32 10TB dask.array<chunksize=(1, 1100, 900), meta=np.ndarray>
    crs              float64 8B ...
Attributes: (12/13)
    Author:                            Harald Rybka, Katharina Lengfeld
    Conventions:                       CF-1.6
    history:                           Created at 2021-07-09 09:10:06.385653
    institution:                       Deutscher Wetterdienst (DWD)
    reference:                         10.5676/DWD/RADKLIM_YW_V2017.002
    title:                             RADKLIM - radar-based precipitation cl...
    ...                                ...
    mlcast_created_on:                 2026-02-27T12:03:00
    mlcast_created_by:                 Leif Denby <lcd@dmi.dk>
    mlcast_created_with:               https://github.com/mlcast-community/ml...
    mlcast_dataset_version:            0.1.1
    mlcast_dataset_identifier:         DE-DWD-radar_precipitation-RADKLIM
    mlcast_dataset_identifier_format:  {country_code}-{entity}-{physical_vari...

Start using the dataset 🙂

Contributing

We are always looking for new datasets to add to the catalog. If you have a dataset you would like to contribute, please open an issue or a pull request.

License

This project is dual-licensed under either:

at your option.

See LICENSE for more details.

Metadata

Release files for mlcast-datasets 0.3.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for mlcast-datasets 0.3.0
File Interpreter ABI Platform Actions
mlcast_datasets-0.3.0-py3-none-any.whl Python 3 none any Details for mlcast_datasets-0.3.0-py3-none-any.whl

Release files / mlcast_datasets-0.3.0-py3-none-any.whl

Download URL mlcast_datasets-0.3.0-py3-none-any.whl
Size 11.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
72d2dfc1b817f13c640657fd0d31a3a37ea889ccdf99ead60fe7d9c0febf9ccf
BLAKE2b-256 checksum
How to use checksums
dcd9294647ee6b1ecf27b18925042fb491f5554bd8c90e3bcebdbf1fb67240e1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.13.12

Security for mlcast-datasets 0.3.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page