Reflective Data Catalog

Reflective's unified Python interface for accessing SAI (Stratospheric Aerosol Injection) climate model data across cloud providers (S3, GCS, Azure, Cloudflare R2), for use on the Reflective Cloud Hub and anywhere else you run Python.

Every dataset is registered in a single packaged YAML catalog and loaded through one consistent xarray-based interface for browsing, searching, and loading SRM-related datasets. All available datasets can be seen here with more information in the Reflective Cloud Hub documentation. We've also included an example Jupyter Notebook showing how to use the tool.

The public ARISE sources work with no credentials at all; sources on the private Reflective hub bucket require AWS credentials (available automatically on the Reflective Cloud Hub). See the credentials matrix below.

Installation

pip install reflective-data-catalog

Optional extras add the Google Cloud CMIP6/GeoMIP catalog (via intake-esm) and ESGF access (via intake-esgf; needs Python >= 3.12) — neither is required for the core catalog:

pip install "reflective-data-catalog[esm]"    # catalog.esm / catalog.geomip_cloud
pip install "reflective-data-catalog[esgf]"   # catalog.esgf
pip install "reflective-data-catalog[esm,esgf]"

For development:

git clone https://github.com/ReflectiveCloud/reflective-data-catalog.git
cd reflective-data-catalog
pip install -e ".[dev]"
pre-commit install

This installs a pre-commit hook that automatically runs Ruff linting (with auto-fix) and formatting on every commit.

Quick Start

from reflective_data_catalog import ReflectiveCatalog

rdc = ReflectiveCatalog()

# Public data — works with no credentials at all
ds = rdc.arise_sai_15(variable="TREFHT", time_frequency="month_1").to_dask()

# Public Zarr stores on Cloudflare R2 — also no credentials. Tables and
# realms are Zarr groups; variables are selected from the opened dataset —
# list_variables() enumerates them (list_variables(table="day") aggregates
# across the day/* groups; add realm= to pin one group).
ds = rdc.cesm2_waccm_g6_1p5k_hilla(table="Amon", realm="atmos_3d", ensemble="r1").to_dask()
temperature = ds["T"]

# Load into memory instead of lazily
ds = rdc.miroc_es2h_g6_1p5k_sai(table="Mon", ensemble="r01").read()
surface_temp = ds["SurfT"]

Credentials

Three access classes cover every source in the catalog:

Missing credentials raise a MissingCredentialsError that names the missing configuration (for example the CLOUDFLARE_R2_ACCOUNT_ID environment variable) rather than a provider stack trace.

Available Sources

All 27 sources come from the packaged catalog file (src/reflective_data_catalog/data-catalog.yaml). Entries marked experimental are still being populated or verified — see docs/migration-matrix.md for the authoritative stability listing.

The ten simulator_* entries are the public simulator-input stores (simulator-inputs/ on the R2 bucket): grouped Zarr stores of monthly (CESM-MA also daily) 2D atmosphere fields with table/realm groups (Mon/atmos_2d default, plus atmos_2d_derived and atmos_2d_tasminmax) and a 3-member ensemble selected exactly like the other public CESM/MIROC entries.

Usage

Selecting Parameters

Every source takes the canonical keyword arguments ensemble, table, and variable, plus per-source extras such as variant, realm, time_frequency, or version. CMIP6-style aliases are accepted permanently: ensemble_member/member_id → ensemble, table_id → table, variable_id → variable.

# Canonical kwargs (UKESM HiLLA is NetCDF: UM stream tables + a time segment)
ds = rdc.ukesm1_g6_1p5k_hilla(
    variable="ua", table="ap4", time="AERmon", ensemble="r12i1p1f2"
).to_dask()

# Aliases work identically
ds = rdc.ukesm1_g6_1p5k_hilla(
    variable_id="ua", table_id="ap4", time="AERmon", member_id="r12i1p1f2"
).to_dask()

# Typos raise a TypeError listing the valid parameters — nothing loads silently
rdc.cesm2_waccm_ssp245(varaible="SALT")
# TypeError: cesm2_waccm_ssp245: unknown parameter(s) ['varaible']. ...

The CESM2-WACCM and MIROC-ES2H entries open grouped public Zarr stores: table and realm select the Zarr group, the ensemble is a dataset dimension (ensemble="all" keeps every member), and variables are picked from the opened dataset. The MIROC entries additionally take variant= — 'baseline' selects the SSP2-4.5 reference store (the default on the HiLLA entry); the experiment name selects the experiment store:

# SSP2-4.5 reference outputs (the default variant on the HiLLA entry)
ds = rdc.miroc_es2h_g6_1p5k_hilla(table="Mon", ensemble="r01").to_dask()
surface_temp = ds["SurfT"]

# The experiments themselves (the HiLLA experiment store uses
# member-suffixed realm groups — use .discover() to list them)
ds = rdc.miroc_es2h_g6_1p5k_hilla(
    variant="G6-1.5K-HiLLA", table="Amon", realm="atmos_2d_r03"
).to_dask()
ds = rdc.miroc_es2h_g6_1p5k_sai().to_dask()   # variant='G6-1.5K-SAI' default

# The full ten-member ensemble as one dataset
ds = rdc.miroc_es2h_g6_1p5k_sai(ensemble="all").to_dask()

Discovering Available Data

Catalog-level listing and search return structured records; pass verbose=True for a printed summary:

records = rdc.list_sources()        # structured records for every source
rdc.list_sources(verbose=True)      # human-readable printout

for rec in rdc.search(term="ukesm"):
    print(rec["name"], rec["kind"])  # every result carries its kind

rdc.list_tags()                      # tags across catalog entries
rdc.get_parameters("arise_sai_15")   # {name, driver, description, parameters}
src = rdc.get_source("arise_sai_15") # string-keyed access to any entry
rdc.help()                           # overview help text

Each source scans cloud storage to report what actually exists (an empty scan returns [] with a warning — never the parameter defaults):

source = rdc.ukesm1_g6_1p5k_hilla()

source.list_variables()   # observed variables (optionally per ensemble/table)
source.list_ensembles()
source.list_tables()

source.discover()         # dict summary: url + ensembles + tables + variables

Errors

The package raises a typed exception hierarchy: CatalogError (bad or unsupported catalog file), SourceNotFoundError (unknown source name; subclasses AttributeError so hasattr and tab-completion keep working), DataNotFoundError (a rendered URL matched no data, or matched it impurely), and MissingCredentialsError (names the missing credential). Calls that use pre-1.0 parameter vocabulary raise errors that carry the old→new guidance from the migration matrix.

Google Cloud CMIP6 / GeoMIP ([esm] extra)

With pip install "reflective-data-catalog[esm]", access cloud-optimized Zarr data from the Google Cloud CMIP6 catalog:

# Search and load in one step
datasets = rdc.esm.load(
    experiment_id=["G6sulfur", "ssp245", "ssp585"],
    variable_id="tas",
    table_id="Amon",
    require_all_on=["source_id", "institution_id"],
)

# Or use the GeoMIP convenience helper
datasets = rdc.geomip_cloud.load_ensemble(
    experiments=["G6sulfur", "ssp245", "ssp585"],
    variable="tas",
)

# Quick single-experiment load
ds_dict = rdc.geomip_cloud.g6sulfur(variable="tas")

# Explore what's available
rdc.geomip_cloud.list_models()
rdc.geomip_cloud.list_variables(experiment_id="G6sulfur")
rdc.geomip_cloud.summary()

# Advanced: direct search then load
subset = rdc.esm.search(
    experiment_id="G6sulfur",
    variable_id=["tas", "pr"],
    table_id="Amon",
)
datasets = subset.to_dataset_dict()

ESGF Data ([esgf] extra)

With pip install "reflective-data-catalog[esgf]", the catalog also provides access to ESGF (Earth System Grid Federation) data:

ds = rdc.esgf.geomip.g6sulfur(model="UKESM1-0-LL", variable="tas")

Migrating from 0.x

Mid-analysis and need the old behavior right now? Pin below 1.0:

pip install "reflective-data-catalog<1"

v1.0 replaces the pre-1.0 dual registration system (flexible source configs plus an intake catalog) with a single packaged YAML catalog and a self-parsed loader. The 8 documented source names are unchanged. What did change:

• One registration mechanism. All sources live in data-catalog.yaml; the flexible-sources system (FlexibleSourceConfig, reflective_data.py) is gone. intake is no longer a runtime dependency — intake-esm/intake-esgf moved behind the [esm]/[esgf] extras.
• Canonical kwargs plus permanent aliases. ensemble/table/variable are canonical; ensemble_member, member_id, table_id, and variable_id are accepted forever.
• Typos now error. Unknown keyword arguments raise TypeError listing the valid parameters. Previously they were silently ignored and the defaults loaded.
• MIROC uses variant=. The old separate MIROC SSP2-4.5 sources were absorbed into the two MIROC entries as variant='baseline' (the default on the HiLLA entry).
• Both UKESM hub entries (ukesm1_ssp245, ukesm1_g6_1p5k_hilla) stay NetCDF in their pre-1.0 stream/time layouts — UM stream table= values (ap4..onm), the time= parameter, and the old defaults all keep working unchanged.
• Parameter value vocabularies changed with the Zarr switches (CESM/MIROC). Old table values (e.g. table='AMON') no longer match the Zarr groups (Amon), and variables are selected from the opened dataset instead of the path. Old values are never silently translated — they raise an error carrying the old→new mapping. The NetCDF-preserving entries (UKESM, E3SM, cesm2_waccm_historical) keep their old vocabularies; E3SM variables stay E3SM-native (T, TREFHT — there is no tas).
• Structured results and typed errors. list_sources()/list_tags()/search() return records (printing behind verbose=True); get_source_config() is replaced by get_source() and get_parameters(); unknown sources raise SourceNotFoundError.

The full old→new table — source by source, kwarg by kwarg, default by default — is in docs/migration-matrix.md, rendered from the machine-readable migration_matrix.yaml that ships inside the package.

Running Tests

Run the full test suite:

pytest

Run with coverage report:

pytest --cov=reflective_data_catalog --cov-report=term-missing

Run a specific test file:

pytest tests/test_catalog.py
pytest tests/test_catalog_entries.py   # instantiates every shipped catalog entry

Tests use the real packaged catalog and mock storage I/O only — no network access or cloud credentials are required.

Requirements

Python >= 3.11 for the core catalog; the [esgf] extra needs Python >= 3.12 (intake-esgf dropped 3.11). Runtime dependencies (and the esm, esgf, and dev extras) are declared in pyproject.toml — that file is the single source of truth for versions.

License

Apache 2.0