Xarray Prism Engine

A multi-format and multi-storage xarray engine with automatic engine detection, and ability to register new data format and uri type for climate data.

Installation

Install via PyPI

pip install xarray-prism

Install via Conda

conda install xarray-prism

Quick Start

Using with xarray

import xarray as xr

# Auto-detect format
ds = xr.open_dataset("my_data.unknown_fmt", engine="prism")

# Remote Zarr on S3
ds = xr.open_dataset(
    "s3://freva/workshop/tas.zarr",
    engine="prism",
    storage_options={
        "anon": True,
        "client_kwargs": {
            "endpoint_url": "https://s3.eu-dkrz-1.dkrz.cloud"
        }
    }
)

# Remote NetCDF3 on S3
ds = xr.open_dataset(
    "s3://freva/workshop/tas.nc",
    engine="prism",
    storage_options={
        "anon": True,
        "client_kwargs": {
            "endpoint_url": "https://s3.eu-dkrz-1.dkrz.cloud"
        }
    }
)

# Remote NetCDF4 on S3
ds = xr.open_dataset(
    "s3://freva/workshop/tas.nc4",
    engine="prism",
    storage_options={
        "anon": True,
        "client_kwargs": {
            "endpoint_url": "https://s3.eu-dkrz-1.dkrz.cloud"
        }
    }
)

# Remote Zarr on S3 - non-anon
ds = xr.open_dataset(
    "s3://bucket/data.zarr",
    engine="prism",
    storage_options={
        "key": "YOUR_KEY",
        "secret": "YOUR_SECRET",
        "client_kwargs": {
            "endpoint_url": "S3_ENDPOINT"
        }
    }
)

# OPeNDAP from THREDDS
ds = xr.open_dataset(
    "https://icdc.cen.uni-hamburg.de/thredds/dodsC/ftpthredds/ar5_sea_level_rise/gia_mean.nc",
    engine="prism"
)

# Local GRIB file
ds = xr.open_dataset("forecast.grib2", engine="prism")

# GeoTIFF
ds = xr.open_dataset("satellite.tif", engine="prism")

# tip: Handle the cache manually by yourself
xr.open_dataset(
    "simplecache::s3://bucket/file.nc3",
    engine="prism",
    storage_options={
        "s3": {"anon": True, "client_kwargs": {"endpoint_url": "..."}},
        "simplecache": {"cache_storage": "/path/to/cache"}
    }
)

# Even for the tif format on the S3 you can pass the credential through
# storage_options which is not supported by rasterio:
xr.open_dataset(
    "s3://bucket/file.tif",
    engine="prism",
    storage_options={
        "key": "YOUR_KEY",
        "secret": "YOUR_SECRET",
        "client_kwargs": {
            "endpoint_url": "S3_ENDPOINT"
        }
    }
)

Supported Formats

Cache management

You can inspect or evict the cache manually:

import xarray_prism as xp

xp.cache_info()
# {'files': 12, 'size_bytes': 2400000000, 'path': '/tmp/xarray-prism-cache'}

# Preview what would be removed
xp.clear_cache(dry_run=True)

# Evict with custom thresholds
xp.clear_cache(max_age_days=3, max_size_gb=2)

# Remove everything
xp.clear_cache(max_age_days=0, max_size_gb=0)

Logging

By default xarray-prism is silent (WARNING level). Set XARRAY_PRISM_LOG_LEVEL to change verbosity:

# Show detection and open steps
XARRAY_PRISM_LOG_LEVEL=DEBUG python my_script.py

# Suppress everything except errors
XARRAY_PRISM_LOG_LEVEL=ERROR python my_script.py

Accepted values: DEBUG, INFO, WARNING, ERROR, CRITICAL.

Customization

Custom Format Detectors and URI Types

You can extend xarray-prism with custom format detectors, URI types, and open handlers by providing a small plugin package. Registration happens at import time, so importing the plugin activates it.

Plugin structure

xarray_prism_myplugin/
  __init__.py   # imports the plugin module (triggers registration)
  plugin.py     # detectors, URI types, and open handlers
pyproject.toml

Plugin implementation

xarray_prism_myplugin/__init__.py

from .plugin import *  # noqa: F401,F403

xarray_prism_myplugin/plugin.py

import xarray as xr
from xarray_prism import register_detector, register_uri_type, registry


@register_uri_type(priority=100)
def detect_myfs_uri(uri: str):
    """Detect a custom filesystem URI."""
    if uri.lower().startswith("myfs://"):
        return "myfs"
    return None


@register_detector(priority=100)
def detect_foo_format(uri: str):
    """Detect a custom file format."""
    if uri.lower().endswith(".foo"):
        return "foo"
    return None


@registry.register("foo", uri_type="myfs")
def open_foo_from_myfs(uri: str, **kwargs):
    """Open .foo files from myfs:// URIs."""
    translated = uri.replace("myfs://", "https://my-gateway.example/")
    return xr.open_dataset(translated, engine="h5netcdf", **kwargs)

Plugin installation

pyproject.toml

[project]
name = "xarray-prism-myplugin"
version = "0.1.0"
dependencies = ["xarray-prism"]

[project.entry-points."xarray_prism.plugins"]
myplugin = "xarray_prism_myplugin"

Using the plugin

After installing the plugin package, import it once to activate the registrations:

import xarray_prism_myplugin  # activates detectors and handlers

import xarray as xr
ds = xr.open_dataset("myfs://bucket/path/data.foo", engine="prism")

Development

Setup Development Environment

# Start test services (MinIO, THREDDS)
docker-compose -f dev-env/docker-compose.yaml up -d --remove-orphans

# Create conda environment
conda create -n xarray-prism python=3.12 -y
conda activate xarray-prism

# Install package in editable mode with dev dependencies
pip install -e ".[dev]"

Running Tests

# Run tests
tox -e test

# Run with coverage
tox -e test-cov

# Lint
tox -e lint

# Type checking
tox -e types

# Auto-format code
tox -e format

Creating a Release

Releases are managed via GitHub Actions and tox:

# Tag a new release (creates git tag)
tox -e release

The release workflow is triggered automatically when:

• A version tag (v*.*.*) is pushed -> Full release to PyPI
• Manual workflow dispatch with RC number -> Pre-release to PyPI