Interface for using cubed with xarray for parallel computation.
Project description
Note: this is a proof-of-concept, and many things are incomplete, untested, or don't work.
cubed-xarray
Interface for using cubed with xarray.
Requirements
- Cubed version >=0.14.2
- Xarray version >=2024.02.0
Installation
Install via pip
pip install cubed-xarray
or conda
conda install -c conda-forge cubed-xarray
Importing
You don't need to import this package in user code. Once poperly installed, xarray should automatically become aware of this package via the magic of entrypoints.
Usage
Xarray objects backed by cubed arrays can be created either by:
- Passing existing
cubed.Array
objects to thedata
argument of xarray constructors, - Calling
.chunk
on xarray objects, - Passing a
chunks
argument toxarray.open_dataset
.
In (2) and (3) the choice to use cubed.Array
instead of dask.array.Array
is made by passing the keyword argument chunked_array_type='cubed'
.
To pass arguments to the constructor of cubed.Array
you should pass them via the dictionary from_array_kwargs
, e.g. from_array_kwargs={'spec': cubed.Spec(allowed_mem='2GB')}
.
If cubed and cubed-xarray are installed but dask is not, then specifying chunked_array_type
is not necessary,
as the entrypoints system will then default to the only chunked parallel backend available (i.e. cubed).
Sharp Edges 🔪
Some things almost certainly won't work yet:
- Certain operations called in xarray but not implemented in cubed, for instance
pad
(see https://github.com/tomwhite/cubed/issues/193) - Array operations involving NaNs - for now use
skipna=True
to avoid eager loading (see https://github.com/pydata/xarray/issues/7243) - Using
parallel=True
withxr.open_mfdataset
won't work because cubed doesn't implement a version ofdask.Delayed
(see https://github.com/pydata/xarray/issues/7810) - Groupby (see https://github.com/tomwhite/cubed/issues/223 and https://github.com/xarray-contrib/flox/issues/224)
xarray.map_blocks
does not actually dispatch tocubed.map_blocks
yet, and will always use Dask.- Certain operations using
cumreduction
(e.g.ffill
andbfill
) are not hooked up to theChunkManager
yet, so will attempt to call dask.
and some other things might work but have not yet been tried:
- Saving to formats other than zarr
In general a bug could take the form of an error, or of a silent attempt to coerce the array type to numpy by immediately computing the underlying array.
Tests
Integration tests for wrapping cubed with xarray also live in this repository.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file cubed-xarray-0.0.7.tar.gz
.
File metadata
- Download URL: cubed-xarray-0.0.7.tar.gz
- Upload date:
- Size: 19.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.0.0 CPython/3.9.16
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | b08a969656c9c05c7168f9eee78911b73d6ab31781c317984b490c8e9d984eac |
|
MD5 | b938161ff5abb78ac9f08c29d50835a4 |
|
BLAKE2b-256 | 013cb5dd75d70fb05e50771148296c20d415e7f20e1c9d86e90e106677ea4388 |
File details
Details for the file cubed_xarray-0.0.7-py3-none-any.whl
.
File metadata
- Download URL: cubed_xarray-0.0.7-py3-none-any.whl
- Upload date:
- Size: 16.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.0.0 CPython/3.9.16
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 83aad6439c4ab5b58601f2ef806e45aac64adc43a84ddec21c5507ec13c8af35 |
|
MD5 | f26a68202947c28bdd3b39d72d6057a5 |
|
BLAKE2b-256 | e7bfd3e75275497bc0a84f3420c15952dd63beb0b2b6cb9fedbad32a640f0750 |