XRPatcher - A ML-Oriented Generic Patcher for xarray Data Structures

Installation |Examples

• J. Emmanuel Johnson
• Quentin Febvre

About

This is a lightweight package to create patch items from xarray data structures. This makes it more compatible with machine learning datasets and dataloaders like PyTorch or TensorFlow. The user simply needs to define the patch dimensions and the stride dimensions and you are good to go! It also reconstructs (or unpatchifies) from arbitrary patches which allows for more robust inference procedures, e.g. to account for border effects from CNN models.

⏩ Examples

Quick Example

import xarray as xr
from xrpatcher import XRDAPatcher

# load demo dataset
data = xr.tutorial.load_dataset("eraint_uvz")

# extract demo dataarray
data = data.u[..., :240, :360]

# Instantiate the patching logic for training
patches = dict(longitude=30, latitude=30)
train_patcher = XRDAPatcher(
    da=data,
    patches=patches,
    strides=patches,        # No Overlap
    check_full_scan=True    # check no extra dimensions
)

# Instantiate the patching logic for testing
patches = dict(longitude=30, latitude=30)
strides = dict(longitude=5, latitude=5)

test_patcher = XRDAPatcher(
    da=data,
    patches=patches,
    strides=strides,        # Overlap
    check_full_scan=True    # check no extra dimensions
)

Extended Example

Example 1: Patching Crash Course

We have an extended example where we demonstrate some of the ways to do the reconstruction!

Example 2: PyTorch Integration

We have an extended example where we demonstrate some nifty PyTorch Integration.

🛠️ Installation

pip

We can directly install it via pip from the

pip install "git+https://github.com/jejjohnson/xrpatcher.git"

Cloning

We can also clone the git repository

git clone https://github.com/jejjohnson/xrpatcher.git
cd xrpatcher

Conda Environment (RECOMMENDED)

We use conda/mamba as our package manager. To install from the provided environment files run the following command.

mamba env create -n environment.yaml

poetry

The easiest way to get started is to simply use the poetry package which installs all necessary dev packages as well

poetry install

pip

We can also install via pip as well

pip install .

Inspiration

There are a few other packages that gave us inspiration for this.

• xbatcher
• PatchExtractor (scikit-learn)