rustytree — Rust-backed xarray DataTree backend

CI PyPI version Python versions License

rustytree registers as xr.open_datatree(engine="rustytree") and walks Zarr v3 hierarchies — both icechunk-backed and vanilla — concurrently in async across one FFI crossing. The goal is a drop-in faster replacement for xr.open_datatree(engine="zarr"), especially on icechunk repos served from object storage where xarray's current per-group sequential decoding is the dominant cost.

Status

Alpha. xr.open_datatree(..., engine="rustytree") works end-to-end against both icechunk repositories (pass session.store) and vanilla Zarr v3 stores (path or s3:// URL). Lazy chunk reads via RustyBackendArray, CF decoding (incl. metadata-only datetime dtype inference), the recursive multi-node walk, glob group= filtering, and non-recursive single-Dataset opens are all in. See CHANGELOG.md for the per-PR breakdown.

Compatibility

Surface Supported Not supported
Zarr format v3 only v2 — passing zarr_format=2 or consolidated=True raises NotImplementedError; use stock engine="zarr" for v2 stores
icechunk icechunk>=2.0 (the current major) older icechunk releases
Python 3.12, 3.13 older versions
Platforms manylinux x86_64, manylinux aarch64, macOS arm64 (wheels); other platforms via sdist Windows / macOS Intel / linux musl wheels (build from source via sdist)

Install

Install from PyPI (the import name stays rustytree; the distribution name on PyPI is rustytree-xarray because rustytree collides with an unrelated dormant package):

pip install rustytree-xarray

Or install from source per Quick start below.

Example

Open the public anonymous NEXRAD KLOT icechunk repo on S3:

import icechunk
import xarray as xr

storage = icechunk.s3_storage(
    bucket="nexrad-arco", prefix="KLOT",
    region="us-east-1", anonymous=True,
)
session = icechunk.Repository.open(storage).readonly_session("main")

# Open the full DataTree (107 nodes — every VCP × sweep combination).
dt = xr.open_datatree(session.store, engine="rustytree")

# Or grab one specific sweep as a flat Dataset (skips walking siblings):
ds = xr.open_dataset(session.store, engine="rustytree",
                     group="/VCP-12/sweep_0")

# Or apply a glob pattern. "Give me sweep_0 from every VCP" is the
# canonical radar workflow — returns a tree filtered to those matches,
# with the VCP container groups auto-included as ancestors.
sweeps_0 = xr.open_datatree(session.store, engine="rustytree",
                            group_filter="/*/sweep_0")

engine="rustytree" is a drop-in replacement for engine="zarr" — same xr.open_datatree / xr.open_dataset entry points, same storage_options / decode_* kwargs. The full demo (with side-by-side timings against engine="zarr") is in notebooks/klot_demo.ipynb.

Quick start

rustytree is a Rust extension built via maturin; uv sync handles the venv, Python deps, and the Rust build (via maturin's PEP 517 hook) in one command.

# Clone + enter the repo
git clone https://github.com/aladinor/rustytree.git
cd rustytree

# Install everything: venv, deps, and the compiled Rust extension.
uv sync --extra dev

# Confirm the install
.venv/bin/pytest tests/

# Launch Jupyter and open the demo notebook
.venv/bin/jupyter lab notebooks/klot_demo.ipynb

The notebooks and tests don't require AWS credentials — KLOT is read anonymously via icechunk's anonymous S3 path. Network speed determines cold-cache timings (expect ~1–3 s for engine="rustytree" on KLOT from a home connection; ~50 s for engine="zarr" on the same).

Documentation

License

Apache License 2.0 (Apache-2.0).

Acknowledgements

Built on zarrs + zarrs_icechunk + icechunk. Sibling project radish proved the PyO3 + xarray-backend pattern. The bottlenecks this project sets out to collapse are tracked in xarray PRs #10742, #11304, and #11302.

Project links

Data verified by PyPI on Jul 26, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for atmoscale from gravatar.com atmoscale

Credits

Author: Alfonso Ladino

GitHub Statistics

Data verified by PyPI on Jul 26, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache Software License (Apache-2.0)

Requires

Python >=3.12

Provides Extra

dev

Tags

xarray zarr datatree rust object-storage

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

rustytree_xarray-0.5.0.tar.gz (1.2 MB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

rustytree_xarray-0.5.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.17+ x86-64

rustytree_xarray-0.5.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (11.6 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.17+ ARM64

rustytree_xarray-0.5.0-cp313-cp313-macosx_11_0_arm64.whl (10.6 MB view details)

Uploaded CPython 3.13macOS 11.0+ ARM64

rustytree_xarray-0.5.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ x86-64

rustytree_xarray-0.5.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (11.6 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ ARM64

rustytree_xarray-0.5.0-cp312-cp312-macosx_11_0_arm64.whl (10.6 MB view details)

Uploaded CPython 3.12macOS 11.0+ ARM64

File details

Details for the file rustytree_xarray-0.5.0.tar.gz.

File metadata

  • Download URL: rustytree_xarray-0.5.0.tar.gz
  • Upload date:
  • Size: 1.2 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for rustytree_xarray-0.5.0.tar.gz
Algorithm Hash digest
SHA256 fc92b58704d1002f6859dcd0391a1bd00c2c848b32127b8b4ec8c56509966278
MD5 8175dd6773f9d859af1c0f2351925771
BLAKE2b-256 10b88d6bdef89d4b93e03cb5302dda03d242bdc6b25248c48883b30a5337d2eb

See more details on using hashes here.

Provenance

The following attestation bundles were made for rustytree_xarray-0.5.0.tar.gz:

Publisher: release.yml on aladinor/rustytree

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file rustytree_xarray-0.5.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for rustytree_xarray-0.5.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 edc0e1eaab444d0d139650448985c58df7f3edd356d368f19ae6ba2b564a2788
MD5 f6b79d9a8ac86b0dc79ad1f35cf3d214
BLAKE2b-256 a3d42169d22068ea9d750288cc30c5c383d3305f724d7c7227c64e6d8af6c688

See more details on using hashes here.

Provenance

The following attestation bundles were made for rustytree_xarray-0.5.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on aladinor/rustytree

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file rustytree_xarray-0.5.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for rustytree_xarray-0.5.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 fc714a7911bc73d62b879de0bc6908a574e8a440bccdf07ab46c66fd23d60169
MD5 9bbfb3e03d1ffbd925245bb2166d6c05
BLAKE2b-256 74d1d03e1fb58b64a33d59eece2666ee342ff8eb6860ebf92d544d30a865fcf6

See more details on using hashes here.

Provenance

The following attestation bundles were made for rustytree_xarray-0.5.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on aladinor/rustytree

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file rustytree_xarray-0.5.0-cp313-cp313-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for rustytree_xarray-0.5.0-cp313-cp313-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 31844cc4b13969858c8ea32ebc648ca31b35bb2899b10bf5c04b48581a614dc6
MD5 55c8f2a682f273473e386db4edd224f3
BLAKE2b-256 30378fb45003f9d9a4d2b40cccd86e67f2fe6ab8db6fc93d7acb788e781ab2ac

See more details on using hashes here.

Provenance

The following attestation bundles were made for rustytree_xarray-0.5.0-cp313-cp313-macosx_11_0_arm64.whl:

Publisher: release.yml on aladinor/rustytree

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file rustytree_xarray-0.5.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for rustytree_xarray-0.5.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 9557bb8929e0f98dc89b6c7ff58349adf4730e2bbe7961ec2752e32ca747bbf6
MD5 0a3a1098ef86b33f0cc10b8cda084d47
BLAKE2b-256 77c799184b6613b80559c63e8f93fce8d132c12df2a8b97c9d4e5ba7df4cd3a5

See more details on using hashes here.

Provenance

The following attestation bundles were made for rustytree_xarray-0.5.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on aladinor/rustytree

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file rustytree_xarray-0.5.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for rustytree_xarray-0.5.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 647d0d49dca7254f56e2538173fdc092a293762291a7ef732aa50bdd840c755e
MD5 aa36114af9bd43ca4557b33b3e1180ff
BLAKE2b-256 4f2de643e58cf4b9818e7c04623a4c7758199a9589e0b424546e8120f5852cec

See more details on using hashes here.

Provenance

The following attestation bundles were made for rustytree_xarray-0.5.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on aladinor/rustytree

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file rustytree_xarray-0.5.0-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for rustytree_xarray-0.5.0-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 f1d9a20723fd0e83f406f0d700173458729289d026188689433d8099249443d1
MD5 ab50fb9663141725278ee93169a14570
BLAKE2b-256 70d0ec54cd7fafb833f9d9627b7434f0c67b2fec279bd0313839b6ddea800b93

See more details on using hashes here.

Provenance

The following attestation bundles were made for rustytree_xarray-0.5.0-cp312-cp312-macosx_11_0_arm64.whl:

Publisher: release.yml on aladinor/rustytree

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.5.0 This release

7 files

0.4.0

7 files

0.3.0

7 files

0.2.1

7 files

0.2.0

7 files

0.1.0

7 files

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page