Xarray-Beam
Xarray-Beam is a Python library for building Apache Beam pipelines with Xarray datasets.
The project aims to facilitate data transformations and analysis on large-scale multi-dimensional labeled arrays, such as:
- Ad-hoc computation on Xarray data, by dividing a
xarray.Datasetinto many smaller pieces ("chunks"). - Adjusting array chunks, using the Rechunker algorithm.
- Ingesting large, multi-dimensional array datasets into an analysis-ready, cloud-optimized format, namely Zarr (see also Pangeo Forge).
- Calculating statistics (e.g., "climatology") across distributed datasets with arbitrary groups.
For more about our approach and how to get started, read the documentation!
Warning: Xarray-Beam is a sharp tool 🔪
Xarray-Beam is relatively new, and focused on expert users:
- We use it extensively at Google for processing large-scale weather datasets, but there is not yet a vibrant external community.
- It provides low-level abstractions that facilitate writing very large scale data pipelines (e.g., 100+ TB), but by design it requires explicitly thinking about how every operation is parallelized.
Installation
Xarray-Beam requires recent versions of immutabledict, Xarray, Dask, Rechunker, Zarr, and Apache Beam. For best performance when writing Zarr files, use Xarray 0.19.0 or later.
Disclaimer
Xarray-Beam is an experiment that we are sharing with the outside world in the hope that it will be useful. It is not a supported Google product. We welcome feedback, bug reports and code contributions, but cannot guarantee they will be addressed.
See the "Contribution guidelines" for more.
Credits
Contributors:
- Stephan Hoyer
- Jason Hickey
- Cenk Gazen
- Alex Merose
Metadata
Release files for xarray-beam 0.11.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| xarray_beam-0.11.5.tar.gz | 75.9 kB | Without provenance | Details for xarray_beam-0.11.5.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| xarray_beam-0.11.5-py3-none-any.whl | Python 3 | none | any | Details for xarray_beam-0.11.5-py3-none-any.whl |
Total release size: 164.3 kB
Release files / xarray_beam-0.11.5.tar.gz
| Download URL | xarray_beam-0.11.5.tar.gz |
|---|---|
| Size | 75.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1a1cdafbbb4f539f7062a4a0350a3ad28f4c5fd2488b41cddd8e079bd77d9566
|
|
BLAKE2b-256 checksum How to use checksums |
807d2e021815215a37830883a8bb29b4afe803e1baecbfa592b08b889f60ae3c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.14
|
Release files / xarray_beam-0.11.5-py3-none-any.whl
| Download URL | xarray_beam-0.11.5-py3-none-any.whl |
|---|---|
| Size | 88.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
af280edc084ebd81b56d10827565fb8d0d5cb021e6f8050eea5e2e5798272af5
|
|
BLAKE2b-256 checksum How to use checksums |
37bcbf4de0cfe8ccd90554c3fe4b2f493af1d6f5fdd53f9e652a7116aa134ce2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.14
|
Security for xarray-beam 0.11.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.