rustytree — Rust-backed xarray DataTree backend

CI PyPI version Python versions License

rustytree registers as xr.open_datatree(engine="rustytree") and walks Zarr v3 hierarchies — both icechunk-backed and vanilla — concurrently in async across one FFI crossing. The goal is a drop-in faster replacement for xr.open_datatree(engine="zarr"), especially on icechunk repos served from object storage where xarray's current per-group sequential decoding is the dominant cost.

Status

Alpha. xr.open_datatree(..., engine="rustytree") works end-to-end against both icechunk repositories (pass session.store) and vanilla Zarr v3 stores (path or s3:// URL). Lazy chunk reads via RustyBackendArray, CF decoding (incl. metadata-only datetime dtype inference), the recursive multi-node walk, glob group= filtering, and non-recursive single-Dataset opens are all in. See CHANGELOG.md for the per-PR breakdown.

Compatibility

Surface Supported Not supported
Zarr format v3 only v2 — passing zarr_format=2 or consolidated=True raises NotImplementedError; use stock engine="zarr" for v2 stores
icechunk icechunk>=2.0 (the current major) older icechunk releases
Python 3.12, 3.13 older versions
Platforms manylinux x86_64, manylinux aarch64, macOS arm64 (wheels); other platforms via sdist Windows / macOS Intel / linux musl wheels (build from source via sdist)

Install

Install from PyPI (the import name stays rustytree; the distribution name on PyPI is rustytree-xarray because rustytree collides with an unrelated dormant package):

pip install rustytree-xarray

Or install from source per Quick start below.

Example

Open the public anonymous NEXRAD KLOT icechunk repo on S3:

import icechunk
import xarray as xr

storage = icechunk.s3_storage(
    bucket="nexrad-arco", prefix="KLOT",
    region="us-east-1", anonymous=True,
)
session = icechunk.Repository.open(storage).readonly_session("main")

# Open the full DataTree (107 nodes — every VCP × sweep combination).
dt = xr.open_datatree(session.store, engine="rustytree")

# Or grab one specific sweep as a flat Dataset (skips walking siblings):
ds = xr.open_dataset(session.store, engine="rustytree",
                     group="/VCP-12/sweep_0")

# Or apply a glob pattern. "Give me sweep_0 from every VCP" is the
# canonical radar workflow — returns a tree filtered to those matches,
# with the VCP container groups auto-included as ancestors.
sweeps_0 = xr.open_datatree(session.store, engine="rustytree",
                            group_filter="/*/sweep_0")

engine="rustytree" is a drop-in replacement for engine="zarr" — same xr.open_datatree / xr.open_dataset entry points, same storage_options / decode_* kwargs. The full demo (with side-by-side timings against engine="zarr") is in notebooks/klot_demo.ipynb.

Quick start

rustytree is a Rust extension built via maturin; uv sync handles the venv, Python deps, and the Rust build (via maturin's PEP 517 hook) in one command.

# Clone + enter the repo
git clone https://github.com/aladinor/rustytree.git
cd rustytree

# Install everything: venv, deps, and the compiled Rust extension.
uv sync --extra dev

# Confirm the install
.venv/bin/pytest tests/

# Launch Jupyter and open the demo notebook
.venv/bin/jupyter lab notebooks/klot_demo.ipynb

The notebooks and tests don't require AWS credentials — KLOT is read anonymously via icechunk's anonymous S3 path. Network speed determines cold-cache timings (expect ~1–3 s for engine="rustytree" on KLOT from a home connection; ~50 s for engine="zarr" on the same).

Documentation

License

Apache License 2.0 (Apache-2.0).

Acknowledgements

Built on zarrs + zarrs_icechunk + icechunk. Sibling project radish proved the PyO3 + xarray-backend pattern. The bottlenecks this project sets out to collapse are tracked in xarray PRs #10742, #11304, and #11302.

Project links

Data verified by PyPI on Jul 24, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.5.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for atmoscale from gravatar.com atmoscale

Credits

Author: Alfonso Ladino

GitHub Statistics

Data verified by PyPI on Jul 24, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache Software License (Apache-2.0)

Requires

Python >=3.12

Provides Extra

dev

Tags

xarray zarr datatree rust object-storage

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

rustytree_xarray-0.4.0.tar.gz (1.2 MB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

rustytree_xarray-0.4.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.0 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.17+ x86-64

rustytree_xarray-0.4.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (11.4 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.17+ ARM64

rustytree_xarray-0.4.0-cp313-cp313-macosx_11_0_arm64.whl (10.3 MB view details)

Uploaded CPython 3.13macOS 11.0+ ARM64

rustytree_xarray-0.4.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.0 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ x86-64

rustytree_xarray-0.4.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (11.4 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ ARM64

rustytree_xarray-0.4.0-cp312-cp312-macosx_11_0_arm64.whl (10.3 MB view details)

Uploaded CPython 3.12macOS 11.0+ ARM64

File details

Details for the file rustytree_xarray-0.4.0.tar.gz.

File metadata

  • Download URL: rustytree_xarray-0.4.0.tar.gz
  • Upload date:
  • Size: 1.2 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for rustytree_xarray-0.4.0.tar.gz
Algorithm Hash digest
SHA256 b990318490a73c626533028ce8d080af7989193107d4f8c42d3ea994871f9d27
MD5 2fce03cbcf43404aedee686df3c04bee
BLAKE2b-256 e4efdc2fb6c0cbead0bd3883e37ab7148bde8ec5d20ee8cd42dd6718c8887eb5

See more details on using hashes here.

Provenance

The following attestation bundles were made for rustytree_xarray-0.4.0.tar.gz:

Publisher: release.yml on aladinor/rustytree

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file rustytree_xarray-0.4.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for rustytree_xarray-0.4.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 ea54adcfbadce7d33aaa8067e9dfdaa15feef72f0885fbbb398503f3900f9918
MD5 dd48c797ef8201f3d274f3dc1f85628d
BLAKE2b-256 f22357c119ad2c4043b50df2a44b9cd9dcad2553ad92fba24ee4449a8d4f45f9

See more details on using hashes here.

Provenance

The following attestation bundles were made for rustytree_xarray-0.4.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on aladinor/rustytree

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file rustytree_xarray-0.4.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for rustytree_xarray-0.4.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 695761a71c01cb499f2e0cb94386e5b949d383a5a53f508a294b601f29c34cd7
MD5 32c77ef6a4226e64e186530b2b8e01b2
BLAKE2b-256 fd34138ca940a676eac3f5d22589540ae7a4c7d57cbca041393ce1a18fb6c23b

See more details on using hashes here.

Provenance

The following attestation bundles were made for rustytree_xarray-0.4.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on aladinor/rustytree

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file rustytree_xarray-0.4.0-cp313-cp313-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for rustytree_xarray-0.4.0-cp313-cp313-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 629a10b7097b8d9b7f138d21b1083b465f1780285348bb93f17d42bfb4525bc7
MD5 b2ebbcd79f26241a927e328969f8f252
BLAKE2b-256 3fc92127ebcfbd3de980a16928007f35cccfd2e1cb07fa35e65d17f7fcbb78ef

See more details on using hashes here.

Provenance

The following attestation bundles were made for rustytree_xarray-0.4.0-cp313-cp313-macosx_11_0_arm64.whl:

Publisher: release.yml on aladinor/rustytree

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file rustytree_xarray-0.4.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for rustytree_xarray-0.4.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 813491dbe4a4286086c29076ffe0e6a66d1113427e920d3af4a68023dc982af9
MD5 1a858a5e90c416487dda890c43cd6c82
BLAKE2b-256 2f9d335d1bfed296976ee95d52928572cc92f768451f5c0c11fc9b5741196568

See more details on using hashes here.

Provenance

The following attestation bundles were made for rustytree_xarray-0.4.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on aladinor/rustytree

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file rustytree_xarray-0.4.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for rustytree_xarray-0.4.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 e0deb6009e9391daa3fb9eb4ce324f5ed2d8def7006b1948a2880220aa268ed5
MD5 780ca38968bb51eccc8c3eb57c50ec30
BLAKE2b-256 2d6c28575d3f154c0733d25e558e4d8fb620055d48bb625b0edd4749c3cf5d7b

See more details on using hashes here.

Provenance

The following attestation bundles were made for rustytree_xarray-0.4.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on aladinor/rustytree

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file rustytree_xarray-0.4.0-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for rustytree_xarray-0.4.0-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 64e9da103e22ad98d1a250e56458baecd20e40b71f19060f63f29588520b9d35
MD5 eb04c37b9faaa828d3388d15f3589b6c
BLAKE2b-256 a4f22731bfb6ba87148b749591f7f45fd1669684fcb99a881f8d8933ffb4000d

See more details on using hashes here.

Provenance

The following attestation bundles were made for rustytree_xarray-0.4.0-cp312-cp312-macosx_11_0_arm64.whl:

Publisher: release.yml on aladinor/rustytree

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.5.0

7 files

This release

0.4.0 This release

7 files

0.3.0

7 files

0.2.1

7 files

0.2.0

7 files

0.1.0

7 files

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page