lazystack

This package provides a familiar interface for lazily reading scientific image stacks, particularly stacks of x-ray images. It's cross-platform, lightweight, and easy to use. Stacks opened via lazystack are indexable (including NumPy-style fancy indexing), iterable, sliceable, and only load the underlying data into memory when absolutely necessary.

Supported formats

• HDF (via h5py) — datasets must be (num_images, height, width) grayscale images.
• TIFF and other TIFF-family files (via tifffile) — grayscale stacks only; colour/multichannel images and hyperstacks are not supported.
• Hamamatsu DCIMG (.dcimg).
• Hamamatsu HIS (.his).

Adding a format is designed to be straightforward. See Adding a format.

TIFF-family files that store the whole stack as a single 3D image (rather than a sequence of 2D images) require the optional zarr package (pip install zarr).

Installation

$ pip install lazystack

Requires Python 3.10 or later.

Usage

Lazy stacks can be created by passing a path or a list/array of paths to the lazystack function. The underlying data is only materialised via integer indexing or upon being cast to a NumPy array, e.g., via np.asarray() or .asarray().

A lazystack exposes NumPy-like attributes: shape, dtype, ndim, size (total elements), and itemsize (bytes per element). Disk usage is available via image_nbytes (bytes per image) and nbytes (bytes for the whole stack). A one-line summary is available via str(stack) or stack.info.

A simple example:

from lazystack import lazystack

with lazystack("path/to/supported_file.something") as stack:
    # Access familiar NumPy array attributes.
    shape = stack.shape
    nbytes = stack.nbytes

    # Slicing the lazystack produces a view (no images in memory).
    view = stack[0:50]

    # Slicing a view also produces a view.
    another_view = view[0:10, [0, 5, 25], ::-1]

    # Integer indexing materialises an image from the view.
    image = another_view[7]

    # `.asarray()` materialises all the images.
    images = another_view.asarray()

Note that nested spatial indexing of views is currently not supported.

NumPy functions that accept array-likes — such as np.take(stack, ...), np.sum(stack), or arithmetic like stack * 2 — convert the stack to a NumPy array first, reading all of its data into memory. This can be unexpected for large stacks, and is avoided by indexing or slicing the stack lazily instead (e.g., stack[a:b, ...] or stack[[0, 50, 100]]).

It's best to open lazystacks within a context manager, but you can also open and close them manually, e.g.:

images = lazystack("path/to/somehis.his")
# Do some stuff.
# ...
# Don't forget to close!
images.close()

If you're opening HDF files, the desired dataset path (within the HDF file) must also be specified, e.g.:

images = lazystack("path/to/some/hdf5.h5", "path/to/some/dset")

Directories of files can be opened by supplying a list or array of filenames, e.g.:

filenames = sorted(input_path.glob("*.tif"))
with lazystack(filenames) as images:
    # Do some stuff.

Currently, this only supports TIFF files.

lazystack also provides iter_chunks for prefetching and yielding successive chunks along any axis of a lazystack (or regular 3D NumPy-like array), e.g.:

from lazystack import lazystack, iter_chunks

# How many chunks will be prefetched (must be at least 1).
num_prefetch = 1
# Memory allowance for all chunks (in GB), including prefetched chunks.
chunk_size_gb = 1.0
# Axis to chunk over (must be 0, 1, or 2).
axis = 0
# Step along the chunk axis (must be at least 1).
step = 1

with lazystack("path/to/somedcimg.dcimg") as images:
    # `start_idx` and `stop_idx` specify the index bounds of each chunk, useful 
    #   for output.
    for chunk, start_idx, stop_idx in iter_chunks(
        images, 
        chunk_size_gb=chunk_size_gb,
        axis=axis,
        step=step,
        num_prefetch=num_prefetch
    ):
        # Do some stuff.

Engine-level laziness and parallelism are intentionally left to consumers: use iter_chunks() to stream bounded-memory chunks through any function, or hand the stack to a chunked compute library such as dask for lazy arithmetic, reductions, or projections over a stack too large to load.

Contributing

Contributions are very welcome! Don't hesitate to reach out if you have any questions, and feel free to open an issue if you have any feedback or encounter any bugs.

If you're adding a new file format, see Adding a format below for the reader contract and a minimal example.

To contribute, clone the repository and set up the development environment with uv, which installs the project along with its development dependencies (pytest, ruff, zarr):

$ git clone https://github.com/domjurkschat/lazystack.git
$ cd lazystack
$ uv sync

Alternatively, install an editable copy with pip, then add the development tools separately:

$ pip install -e .
$ pip install pytest ruff zarr

Run the tests with:

$ uv run pytest

CI enforces a minimum of 75% overall test coverage on pull requests, and Codecov checks that new or changed lines are covered. Check coverage locally with:

$ uv run pytest --cov=lazystack --cov-report=term-missing

Linting and formatting are enforced by CI and can be run locally with:

$ uv run ruff check
$ uv run ruff format

Adding a format

Every supported format is a Stack subclass that sets four attributes and overrides two methods. Lazy slicing, indexing, materialisation, and resource management are handled generically by View and the base class, so a new reader only needs to describe how images are read from disk.

The contract

Both read methods return materialised NumPy arrays; a lazy View is not acceptable, since these are the hooks that actually read from disk. indices may be unordered, duplicated, or negative, and the returned images must follow the given order. If your backend can't do that directly — e.g. h5py requires non-negative, strictly increasing indices — fall back to per-image reads as HDFStack does.

Minimal example

import numpy as np
from lazystack import Stack


class MyStack(Stack):
    """Lazy reader for the fictional .myformat container."""

    def __init__(self, path):
        # Kept as `_file` so `close()`/the context manager releases it.
        self._file = open(path, "rb")
        # shape is (num_images, height, width).
        self.shape = self._read_header()
        self.dtype = np.dtype(np.uint16)
        self.image_nbytes = np.prod(self.shape[1:]) * self.dtype.itemsize
        self.nbytes = self.image_nbytes * self.shape[0]

    def _get_image(self, index):
        image = self._read_image(index)
        return np.frombuffer(image, self.dtype).reshape(self.shape[1:])

    def _get_images(self, indices):
        # Optimised multi-image retrieval preferred, otherwise something like:
        return np.stack([self._get_image(i) for i in indices])


# Opens the underlying file; the context manager closes `_file` on exit.
with MyStack("path/to/file.myformat") as stack:
    image = stack[0]

What works out of the box

• View-backed slicing and spatial cropping (stack[0:50], stack[:, 100:, 50:-50]), so no image data is read until materialisation.
• Integer, slice, list, array, and boolean-mask indexing.
• asarray() / np.asarray() materialisation.
• iter_chunks() (a module-level helper) for prefetched chunking along any axis.
• Context-manager support, and close() which closes _file.
• shape, dtype, ndim, size, itemsize, image_nbytes, nbytes, and info.

Register it

_detect_format() in src/lazystack/_core.py maps a path to a reader class. Add a branch that returns your class, placing it before the tifffile fallback at the end of the function (otherwise the fallback claims the path):

if Path(path).name.lower().endswith(".myformat"):
    return MyStack

If your reader needs constructor arguments (as HDFStack does for dset_name), extend lazystack() to pass them through. Readers can also be instantiated directly without touching _detect_format.

Test it

Add a writer and an _open_stack branch to tests/conftest.py, then include your format name in FORMATS. The parametrised suites in tests/test_core.py then exercise your reader against the shared indexing, attribute, and chunking tests automatically. This usually satisfies the CI coverage floor described above on its own.

Roadmap

• Other file formats -- see Adding a format to contribute a reader.
• Nested spatial indexing.
• .metadata attribute for all formats.
• Harden test suite.
• Stacks of stacks.
• Colour/multichannel and hyperstack support.