rechunkit

Functions to efficiently rechunk multidimensional arrays

Documentation: https://mullenkamp.github.io/rechunkit/

Source Code: https://github.com/mullenkamp/rechunkit

Introduction

Rechunkit is a Python library for efficiently rechunking multidimensional numpy arrays stored as chunks. It uses a generator-based approach for on-the-fly rechunking without requiring the full target array in memory.

Key Features

• Efficient On-the-Fly Rechunking: Uses Python generators to yield rechunked data without requiring the full target array to be stored in memory.
• Memory-Aware Optimization: Employs a smart scaling algorithm to maximize performance within a user-defined memory limit (max_mem).
• LCM Minimization: Utilizes highly composite numbers for chunk guessing to minimize the Least Common Multiple (LCM) between source and target, significantly reducing redundant reads.
• Flexible Data Access: Supports subset selection (sel) and works with any source that implements a numpy __getitem__ style callable (method or function).
• Source-Aligned Selection Reads: When rechunking a subset (sel), read requests are aligned to source chunk boundaries -- even when the selection offset doesn't fall on a chunk boundary. This allows source functions backed by chunk-based storage (HDF5, Zarr, cfdb) to serve each read from aligned chunks without needing to assemble across boundaries.
• Preprocessing Utilities: Includes tools for estimating ideal chunk shapes, calculating memory requirements, and predicting the number of required read operations.

Installation

pip install rechunkit

Quick Example

import numpy as np
from math import prod
from rechunkit import rechunker

shape = (31, 31, 31)
dtype = np.dtype('int32')
source_data = np.arange(1, prod(shape) + 1, dtype=dtype).reshape(shape)
source = source_data.__getitem__

target = np.zeros(shape, dtype=dtype)
for write_chunk, data in rechunker(source, shape, dtype, (5, 2, 4), (4, 5, 3), max_mem=2000):
    target[write_chunk] = data

assert np.all(source_data == target)

See the documentation for detailed guides, integration examples (h5py, zarr), and the full API reference.

License

This project is licensed under the terms of the Apache Software License 2.0.