rechunkit
Functions to efficiently rechunk multidimensional arrays
Documentation: https://mullenkamp.github.io/rechunkit/
Source Code: https://github.com/mullenkamp/rechunkit
Introduction
Rechunkit is a Python library for efficiently rechunking multidimensional numpy arrays stored as chunks. It uses a generator-based approach for on-the-fly rechunking without requiring the full target array in memory.
Key Features
- Efficient On-the-Fly Rechunking: Uses Python generators to yield rechunked data without requiring the full target array to be stored in memory.
- Memory-Aware Optimization: Employs a smart scaling algorithm to maximize performance within a user-defined memory limit (
max_mem). - LCM Minimization: Utilizes highly composite numbers for chunk guessing to minimize the Least Common Multiple (LCM) between source and target, significantly reducing redundant reads.
- Flexible Data Access: Supports subset selection (
sel) and works with any source that implements a numpy__getitem__style callable (method or function). - Source-Aligned Selection Reads: When rechunking a subset (
sel), read requests are aligned to source chunk boundaries -- even when the selection offset doesn't fall on a chunk boundary. This allows source functions backed by chunk-based storage (HDF5, Zarr, cfdb) to serve each read from aligned chunks without needing to assemble across boundaries. - Preprocessing Utilities: Includes tools for estimating ideal chunk shapes, calculating memory requirements, and predicting the number of required read operations.
Installation
pip install rechunkit
Quick Example
import numpy as np
from math import prod
from rechunkit import rechunker
shape = (31, 31, 31)
dtype = np.dtype('int32')
source_data = np.arange(1, prod(shape) + 1, dtype=dtype).reshape(shape)
source = source_data.__getitem__
target = np.zeros(shape, dtype=dtype)
for write_chunk, data in rechunker(source, shape, dtype, (5, 2, 4), (4, 5, 3), max_mem=2000):
target[write_chunk] = data
assert np.all(source_data == target)
See the documentation for detailed guides, integration examples (h5py, zarr), and the full API reference.
License
This project is licensed under the terms of the Apache Software License 2.0.
Metadata
Release files for rechunkit 0.6.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| rechunkit-0.6.0.tar.gz | 14.9 kB | Without provenance | Details for rechunkit-0.6.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| rechunkit-0.6.0-py3-none-any.whl | Python 3 | none | any | Details for rechunkit-0.6.0-py3-none-any.whl |
Total release size: 29.0 kB
Release files / rechunkit-0.6.0.tar.gz
| Download URL | rechunkit-0.6.0.tar.gz |
|---|---|
| Size | 14.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
dda6fa3e45dbb78a2946b1fffbf42d0f859fdc8cebf840640bae341a3a8f68a6
|
|
BLAKE2b-256 checksum How to use checksums |
13607182e5a8b128929d36a89c9695945705acdec8f2681e13a0c3f8d6adb682
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.8.7
|
Release files / rechunkit-0.6.0-py3-none-any.whl
| Download URL | rechunkit-0.6.0-py3-none-any.whl |
|---|---|
| Size | 14.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e24bbf36ca385c241ed476f6a3f397e140ddcaaf46a490932e1d80e2e7204256
|
|
BLAKE2b-256 checksum How to use checksums |
36079c491661bd559409fc5115919968f6c0faadd914fe55726952d5c1ec6233
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.8.7
|
Security for rechunkit 0.6.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.