Xarray-Beam
Xarray-Beam is a Python library for building Apache Beam pipelines with Xarray datasets.
The project aims to facilitate data transformations and analysis on large-scale multi-dimensional labeled arrays, such as:
- Ad-hoc computation on Xarray data, by dividing a
xarray.Datasetinto many smaller pieces ("chunks"). - Adjusting array chunks, using the Rechunker algorithm.
- Ingesting large, multi-dimensional array datasets into an analysis-ready, cloud-optimized format, namely Zarr (see also Pangeo Forge).
- Calculating statistics (e.g., "climatology") across distributed datasets with arbitrary groups.
For more about our approach and how to get started, read the documentation!
Warning: Xarray-Beam is a sharp tool 🔪
Xarray-Beam is relatively new, and focused on expert users:
- We use it extensively at Google for processing large-scale weather datasets, but there is not yet a vibrant external community.
- It provides low-level abstractions that facilitate writing very large scale data pipelines (e.g., 100+ TB), but by design it requires explicitly thinking about how every operation is parallelized.
Installation
Xarray-Beam requires recent versions of immutabledict, Xarray, Dask, Rechunker, Zarr, and Apache Beam. For best performance when writing Zarr files, use Xarray 0.19.0 or later.
Disclaimer
Xarray-Beam is an experiment that we are sharing with the outside world in the hope that it will be useful. It is not a supported Google product. We welcome feedback, bug reports and code contributions, but cannot guarantee they will be addressed.
See the "Contribution guidelines" for more.
Credits
Contributors:
- Stephan Hoyer
- Jason Hickey
- Cenk Gazen
- Alex Merose
2 maintainers
Metadata
2 maintainers
Release files for xarray-beam 0.11.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| xarray_beam-0.11.5.tar.gz | 75.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| xarray_beam-0.11.5-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 164.3 kB
Release files / xarray_beam-0.11.5.tar.gz
| Download URL | xarray_beam-0.11.5.tar.gz |
|---|---|
| Size | 75.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1a1cdafbbb4f539f7062a4a0350a3ad28f4c5fd2488b41cddd8e079bd77d9566
|
|
BLAKE2b-256 checksum How to use checksums |
807d2e021815215a37830883a8bb29b4afe803e1baecbfa592b08b889f60ae3c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.14
|
Release files / xarray_beam-0.11.5-py3-none-any.whl
| Download URL | xarray_beam-0.11.5-py3-none-any.whl |
|---|---|
| Size | 88.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
af280edc084ebd81b56d10827565fb8d0d5cb021e6f8050eea5e2e5798272af5
|
|
BLAKE2b-256 checksum How to use checksums |
37bcbf4de0cfe8ccd90554c3fe4b2f493af1d6f5fdd53f9e652a7116aa134ce2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.14
|
