Skip to main content

An intake plugin for parsing an Earth System Model (ESM) catalog and loading netCDF files and/or Zarr stores into Xarray datasets.

Project description

Intake-esm

Badges

CI GitHub Workflow Status Code Coverage Status pre-commit.ci status
Docs Documentation Status
Package Conda PyPI
License License
Citation Zenodo

Motivation

Computer simulations of the Earth’s climate and weather generate huge amounts of data. These data are often persisted on HPC systems or in the cloud across multiple data assets of a variety of formats (netCDF, zarr, etc...). Finding, investigating, loading these data assets into compute-ready data containers costs time and effort. The data user needs to know what data sets are available, the attributes describing each data set, before loading a specific data set and analyzing it.

Finding, investigating, loading these assets into data array containers such as xarray can be a daunting task due to the large number of files a user may be interested in. Intake-esm aims to address these issues by providing necessary functionality for searching, discovering, data access/loading.

Overview

intake-esm is a data cataloging utility built on top of intake, pandas, and xarray, and it's pretty awesome!

  • Opening an ESM catalog definition file: An Earth System Model (ESM) catalog file is a JSON file that conforms to the ESM Collection Specification. When provided a link/path to an esm catalog file, intake-esm establishes a link to a database (CSV file) that contains data assets locations and associated metadata (i.e., which experiment, model, the come from). The catalog JSON file can be stored on a local filesystem or can be hosted on a remote server.

    In [1]: import intake
    
    In [2]: import intake_esm
    
    In [3]: cat_url = intake_esm.tutorial.get_url("google_cmip6")
    
    In [4]: cat = intake.open_esm_datastore(cat_url)
    
    In [5]: cat
    Out[5]: <GOOGLE-CMIP6 catalog with 4 dataset(s) from 261 asset(s>
    
  • Search and Discovery: intake-esm provides functionality to execute queries against the catalog:

    In [5]: cat_subset = cat.search(
       ...:     experiment_id=["historical", "ssp585"],
       ...:     table_id="Oyr",
       ...:     variable_id="o2",
       ...:     grid_label="gn",
       ...: )
    
    In [6]: cat_subset
    Out[6]: <GOOGLE-CMIP6 catalog with 4 dataset(s) from 261 asset(s)>
    
  • Access: when the user is satisfied with the results of their query, they can load data assets (netCDF and/or Zarr stores) into xarray datasets:

      In [7]: dset_dict = cat_subset.to_dataset_dict()
    
      --> The keys in the returned dictionary of datasets are constructed as follows:
              'activity_id.institution_id.source_id.experiment_id.table_id.grid_label'
      |███████████████████████████████████████████████████████████████| 100.00% [2/2 00:18<00:00]
    

See documentation for more information.

Installation

Intake-esm can be installed from PyPI with pip:

python -m pip install intake-esm

It is also available from conda-forge for conda installations:

conda install -c conda-forge intake-esm

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

intake_esm-2025.2.3.tar.gz (112.6 kB view details)

Uploaded Source

Built Distribution

intake_esm-2025.2.3-py3-none-any.whl (30.3 kB view details)

Uploaded Python 3

File details

Details for the file intake_esm-2025.2.3.tar.gz.

File metadata

  • Download URL: intake_esm-2025.2.3.tar.gz
  • Upload date:
  • Size: 112.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.8

File hashes

Hashes for intake_esm-2025.2.3.tar.gz
Algorithm Hash digest
SHA256 70c4b6fa999684ae1d3a5c1ff134a9cd0a44c47f62e46c7c35518f18678762fd
MD5 9fe262b856cc57e3c9192b4a0eb0a31a
BLAKE2b-256 b43f8497a7f7b2df8fe77d6de1587da8f126356b9df3e7adac43ee890f3f1d63

See more details on using hashes here.

Provenance

The following attestation bundles were made for intake_esm-2025.2.3.tar.gz:

Publisher: pypi.yml on intake/intake-esm

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file intake_esm-2025.2.3-py3-none-any.whl.

File metadata

  • Download URL: intake_esm-2025.2.3-py3-none-any.whl
  • Upload date:
  • Size: 30.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.8

File hashes

Hashes for intake_esm-2025.2.3-py3-none-any.whl
Algorithm Hash digest
SHA256 c836df7be119806894f98fb86f36697cfd05e49c68c5b2418cb89ed696ac478c
MD5 0635e7c909dac99746ec876942f6c109
BLAKE2b-256 da587727592d0c72bfd3ad83144e53d3c2ff5f1c7fa2f0f42b183405574e32ff

See more details on using hashes here.

Provenance

The following attestation bundles were made for intake_esm-2025.2.3-py3-none-any.whl:

Publisher: pypi.yml on intake/intake-esm

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page