Skip to main content

z5 / z5py

Anaconda-Server Badge test-conda test-pypi docs DOI

The new z5 / z5py v3 release adds support for zarr v3 format and s3. It removes the xtensor dependency and is now available via PyPI, in addition to conda-forge.

C++ library (z5) with python bindings (z5py) for zarr and n5 file formats.

This library supports:

  • Zarr format v2 and v3.
  • The n5 format.
  • Access to zarr files on the filesystem and S3 object storage; n5 is only supported on the file system.

Support for the following compression codecs:

Installation

Conda

Conda packages for the relevant systems and python versions are hosted on conda-forge:

$ conda install -c conda-forge z5py

Pip

Wheels are published on PyPI:

$ pip install z5py

The PyPI wheels are built with all compression codecs but without the S3 backend (z5py.S3File will raise "z5 was not compiled with s3 support"). For S3 support, install via conda or build from source with -DWITH_S3=ON.

From Source

The easiest way to build the library from source is using a conda-environment with all necessary dependencies. You can find the conda environment files for build environments in .environments/unix

To set up the conda environment and install the package on unix:

$ conda env create -f environments/unix/z5-dev.yaml
$ conda activate z5-dev
$ mkdir bld
$ cd bld
$ cmake -DWITH_ZLIB=ON -DWITH_BZIP2=ON -DCMAKE_INSTALL_PREFIX=/path/to/install ..
$ make install

Note that in the CMakeLists.txt, we try to infer the active conda-environment automatically. If this fails, you can set it manually via -DCMAKE_PREFIX_PATH=/path/to/conda-env. To specify where to install the package, set:

  • CMAKE_INSTALL_PREFIX: where to install the C++ headers
  • PYTHON_MODULE_INSTALL_DIR: where to install the python package (set to site-packages of active conda env by default)

If you want to include z5 in another C++ project, note that the library itself is header-only. However, you need to link against the compression codecs that you use.

If you don't want to use conda for dependency management, the following dependencies are necessary:

Examples / Usage

Python

The Python API is very similar to h5py. Some differences are:

  • The constructor of File takes the boolean argument use_zarr_format, which determines whether the zarr or N5 format is used (if set to None, an attempt is made to automatically infer the format).
  • There is no need to close File, hence the with block isn't necessary (but supported).
  • Linked datasets (my_file['new_ds'] = my_file['old_ds']) are not supported
  • Broadcasting is only supported for scalars in Dataset.__setitem__
  • Arbitrary leading and trailing singleton dimensions can be added/removed/rolled through in Dataset.__setitem__
  • Compatibility of exception handling is a goal, but not necessarily guaranteed.
  • Because zarr/N5 are usually used with large data, z5py compresses blocks by default where h5py does not. The default compressors are
    • zarr: "blosc"
    • n5: "gzip"

Some examples:

import z5py
import numpy as np

# create a file and a dataset
f = z5py.File('array.zr', use_zarr_format=True)
ds = f.create_dataset('data', shape=(1000, 1000), chunks=(100, 100), dtype='float32')

# write array to a roi
x = np.random.random_sample(size=(500, 500)).astype('float32')
ds[:500, :500] = x

# broadcast a scalar to a roi
ds[500:, 500:] = 42.

# read array from a roi
y = ds[250:750, 250:750]

# create a group and create a dataset in the group
g = f.create_group('local_group')
g.create_dataset('local_data', shape=(100, 100), chunks=(10, 10), dtype='uint32')

# open dataset from group or file
ds_local1 = f['local_group/local_data']
ds_local2 = g['local_data']

# read and write attributes
attributes = ds.attrs
attributes['foo'] = 'bar'
baz = attributes['foo']

C++

Z5 aims to supports different storage implementations. The default is to use the filesystem, implementations to also supports AWS-S3 and Google Cloud Storage are work in progress. The API implements factory functions like createFile or createDataset in the factory header. These functions need to be called with the corresponding handle, like z5::filesystem::handle::File or z5::s3::handle::File in order to specify which backend to use.

The library is intended to be used with an in-memory multiarray that holds the data. Data is passed in and out via a lightweight, non-owning strided view, z5::multiarray::ArrayView / ConstArrayView (a data pointer plus shape and element strides, compatible with numpy arrays), see implementation and the IO functions readSubarray / writeSubarray in array_access.hxx. To interface with another multiarray implementation, wrap its buffer in an ArrayView (data pointer + shape + element strides).

Some examples:

#include "json.hpp"

// factory functions to create files, groups and datasets
#include "z5/factory.hxx"
// handles for z5 filesystem objects
#include "z5/filesystem/handle.hxx"
// strided-view io for multi-arrays
#include "z5/multiarray/array_access.hxx"
// attribute functionality
#include "z5/attributes.hxx"

int main() {

  // get handle to a File on the filesystem
  z5::filesystem::handle::File f("data.zr");
  // if you wanted to use a different backend, for example AWS, you
  // would need to use this instead:
  // z5::s3::handle::File f;

  // create the file in zarr format
  const bool createAsZarr = true;
  z5::createFile(f, createAsZarr);

  // create a new zarr dataset
  const std::string dsName = "data";
  std::vector<size_t> shape = { 1000, 1000, 1000 };
  std::vector<size_t> chunks = { 100, 100, 100 };
  auto ds = z5::createDataset(f, dsName, "float32", shape, chunks);

  // write array to roi; the data lives in a C-contiguous buffer that we
  // wrap in a (non-owning) strided view
  z5::types::ShapeType offset1 = { 50, 100, 150 };
  z5::types::ShapeType shape1 = { 150, 200, 100 };
  std::vector<float> buffer1(150 * 200 * 100, 42.0);
  z5::multiarray::ConstArrayView<float> array1(buffer1.data(), shape1,
                                               z5::multiarray::cOrderStrides(shape1));
  z5::multiarray::writeSubarray<float>(ds, array1, offset1.begin());

  // read array from roi (values that were not written before are filled with a fill-value)
  z5::types::ShapeType offset2 = { 100, 100, 100 };
  z5::types::ShapeType shape2 = { 300, 200, 75 };
  std::vector<float> buffer2(300 * 200 * 75);
  z5::multiarray::ArrayView<float> array2(buffer2.data(), shape2,
                                          z5::multiarray::cOrderStrides(shape2));
  z5::multiarray::readSubarray<float>(ds, array2, offset2.begin());

  // get handle for the dataset
  const auto dsHandle = z5::filesystem::handle::Dataset(f, dsName);

  // read and write json attributes
  nlohmann::json attributesIn;
  attributesIn["bar"] = "foo";
  attributesIn["pi"] = 3.141593;
  z5::writeAttributes(dsHandle, attributesIn);

  nlohmann::json attributesOut;
  z5::readAttributes(dsHandle, attributesOut);
  
  return 0;
}

C

There are external efforts to implement a C-Api wrapper for z5. You can check it out here.

R

There exists a prototype by @gdkrmr to provide R bindings for z5. It is still in an early stage, but looks very promising.

Citation

If you use this library in your research, please cite it via the associated DOI:

@misc{pape_z5_2019,
  doi = {10.5281/ZENODO.3585752},
  url = {https://zenodo.org/record/3585752},
  author = {Pape,  Constantin},
  title = {constantinpape/z5},
  publisher = {Zenodo},
  year = {2019}
}

Current Limitations / TODOs

  • No thread / process synchronization -> writing to the same chunk in parallel will lead to undefined behavior.
  • Supports only little endianness and C-order for the zarr format.

A note on axis ordering

Internally, n5 uses column-major (i.e. x, y, z) axis ordering, while z5 uses row-major (i.e. z, y, x). While this is mostly handled internally, it means that the metadata does not transfer 1 to 1, but needs to be reversed for most shapes. Concretely:

n5 z5
Shape s_x, s_y, s_z s_z, s_y, s_x
Chunk-Shape c_x, c_y, c_z c_z, c_y, c_x
Chunk-Ids i_x, i_y, i_z i_z, i_y, i_x

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

z5py-3.0.2.tar.gz (390.2 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

z5py-3.0.2-cp314-cp314-win_amd64.whl (913.3 kB view details)

Uploaded CPython 3.14Windows x86-64

z5py-3.0.2-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (1.7 MB view details)

Uploaded CPython 3.14manylinux: glibc 2.27+ x86-64manylinux: glibc 2.28+ x86-64

z5py-3.0.2-cp314-cp314-macosx_11_0_arm64.whl (1.1 MB view details)

Uploaded CPython 3.14macOS 11.0+ ARM64

z5py-3.0.2-cp313-cp313-win_amd64.whl (890.2 kB view details)

Uploaded CPython 3.13Windows x86-64

z5py-3.0.2-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (1.7 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.27+ x86-64manylinux: glibc 2.28+ x86-64

z5py-3.0.2-cp313-cp313-macosx_11_0_arm64.whl (1.1 MB view details)

Uploaded CPython 3.13macOS 11.0+ ARM64

z5py-3.0.2-cp312-cp312-win_amd64.whl (890.4 kB view details)

Uploaded CPython 3.12Windows x86-64

z5py-3.0.2-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (1.7 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.27+ x86-64manylinux: glibc 2.28+ x86-64

z5py-3.0.2-cp312-cp312-macosx_11_0_arm64.whl (1.1 MB view details)

Uploaded CPython 3.12macOS 11.0+ ARM64

z5py-3.0.2-cp311-cp311-win_amd64.whl (891.5 kB view details)

Uploaded CPython 3.11Windows x86-64

z5py-3.0.2-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (1.7 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.27+ x86-64manylinux: glibc 2.28+ x86-64

z5py-3.0.2-cp311-cp311-macosx_11_0_arm64.whl (1.1 MB view details)

Uploaded CPython 3.11macOS 11.0+ ARM64

File details

Details for the file z5py-3.0.2.tar.gz.

File metadata

  • Download URL: z5py-3.0.2.tar.gz
  • Upload date:
  • Size: 390.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for z5py-3.0.2.tar.gz
Algorithm Hash digest
SHA256 8a5872c5113321441397cf81ec44234257fe8b94a8e0c8f48f1c997718af3aae
MD5 b47d6ce9dd764beda0f527a986fccacd
BLAKE2b-256 ba8499a3091146541781c4f2072d7dff1e62b6b8a5793ebd7bf9da1e597f2343

See more details on using hashes here.

Provenance

The following attestation bundles were made for z5py-3.0.2.tar.gz:

Publisher: wheels.yml on constantinpape/z5

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file z5py-3.0.2-cp314-cp314-win_amd64.whl.

File metadata

  • Download URL: z5py-3.0.2-cp314-cp314-win_amd64.whl
  • Upload date:
  • Size: 913.3 kB
  • Tags: CPython 3.14, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for z5py-3.0.2-cp314-cp314-win_amd64.whl
Algorithm Hash digest
SHA256 cc61a30b4ab7f99aa5b719c5fd8d330b7fe28a93d58c5908083ab355f679dac0
MD5 fbf3f3596c7bd9e2308df489a7831fa9
BLAKE2b-256 d1440ac7e6dad86aeaaa766319b06ee8bdd5d69b680f2c54fcb77aaf0bebc5c5

See more details on using hashes here.

Provenance

The following attestation bundles were made for z5py-3.0.2-cp314-cp314-win_amd64.whl:

Publisher: wheels.yml on constantinpape/z5

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file z5py-3.0.2-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for z5py-3.0.2-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 43c3fa6a89e58138035c9d9471caf58290bbdb006c3623677ee28da36768ea25
MD5 7a7dc3b52d9c5b403c4e48507f859c7c
BLAKE2b-256 37d30afc8ad90e4b0cee452529fbe83570938071d70aef591ed0f27a391ea1e2

See more details on using hashes here.

Provenance

The following attestation bundles were made for z5py-3.0.2-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl:

Publisher: wheels.yml on constantinpape/z5

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file z5py-3.0.2-cp314-cp314-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for z5py-3.0.2-cp314-cp314-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 95fcff57468cd4856945f2f167955788e945cfe2c6322d7f3f7bd604ed9a7222
MD5 69f7de582acd82c578ca445aad20f66a
BLAKE2b-256 bd2466529fc35fc876b6ef6c96e66003e833395e040e8341e32d692a504e6452

See more details on using hashes here.

Provenance

The following attestation bundles were made for z5py-3.0.2-cp314-cp314-macosx_11_0_arm64.whl:

Publisher: wheels.yml on constantinpape/z5

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file z5py-3.0.2-cp313-cp313-win_amd64.whl.

File metadata

  • Download URL: z5py-3.0.2-cp313-cp313-win_amd64.whl
  • Upload date:
  • Size: 890.2 kB
  • Tags: CPython 3.13, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for z5py-3.0.2-cp313-cp313-win_amd64.whl
Algorithm Hash digest
SHA256 2ec022f4be918d9a1e6ee0eb05a50978b06bc58a8aa7a7052b22a13190d3cf7e
MD5 4683ef041210e98dba28dec81fc8cacb
BLAKE2b-256 5ae566e540425d4b130a4ba455cd9ade93fe1d3b7a0995848cce540fb8f8e517

See more details on using hashes here.

Provenance

The following attestation bundles were made for z5py-3.0.2-cp313-cp313-win_amd64.whl:

Publisher: wheels.yml on constantinpape/z5

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file z5py-3.0.2-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for z5py-3.0.2-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 269a78740e862a71743ee36c060205fcece34fc268bf9a40f6db445ec3fa8ef0
MD5 98d40c11a3f028da1fa30d0e76be6189
BLAKE2b-256 96e882105eb4d63f2f71d09a8104fd0de493fcd43b4f6d61861d496da2897ff9

See more details on using hashes here.

Provenance

The following attestation bundles were made for z5py-3.0.2-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl:

Publisher: wheels.yml on constantinpape/z5

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file z5py-3.0.2-cp313-cp313-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for z5py-3.0.2-cp313-cp313-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 54f50eb829c2cabdcc83afa708d6d8c102747c704d67909233502b0bff2a2a67
MD5 4a0702260e0281fe86289b9933920452
BLAKE2b-256 b30adc0ec179ab997d8e0bb3246f4d7e11933e0de34f65b1eaf3b91e79ac6eec

See more details on using hashes here.

Provenance

The following attestation bundles were made for z5py-3.0.2-cp313-cp313-macosx_11_0_arm64.whl:

Publisher: wheels.yml on constantinpape/z5

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file z5py-3.0.2-cp312-cp312-win_amd64.whl.

File metadata

  • Download URL: z5py-3.0.2-cp312-cp312-win_amd64.whl
  • Upload date:
  • Size: 890.4 kB
  • Tags: CPython 3.12, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for z5py-3.0.2-cp312-cp312-win_amd64.whl
Algorithm Hash digest
SHA256 f44d2da0a6afdc3b9a6f16bc01b46fcf10f248389c51d9ef67284a488cff5e66
MD5 5e484a10166a3eddd52115ca6d53fd2b
BLAKE2b-256 67590fd4d8e5bd4e6ef895c397d4d5f817016d2426031e9123643172475cf16c

See more details on using hashes here.

Provenance

The following attestation bundles were made for z5py-3.0.2-cp312-cp312-win_amd64.whl:

Publisher: wheels.yml on constantinpape/z5

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file z5py-3.0.2-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for z5py-3.0.2-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 d14d66e328c5951750599a4486a00486397ddc3c27cdd33a57f20708ecfced4d
MD5 6ddcb11bb052e21380d3b44e74b0638b
BLAKE2b-256 b15dd889339a038661d6bcf28590d500f0b18b767a27aec60596fb91de968112

See more details on using hashes here.

Provenance

The following attestation bundles were made for z5py-3.0.2-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl:

Publisher: wheels.yml on constantinpape/z5

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file z5py-3.0.2-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for z5py-3.0.2-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 45035a3c0c4c97b214fd773d66fd8e77d724994e30eaa75510205e242957391f
MD5 e8fe1937eda541d845df88804f7677ba
BLAKE2b-256 cd055b1618ef06577913fbbda9c1a951c881de1f42ccf3b3be8c6c5dc850f93e

See more details on using hashes here.

Provenance

The following attestation bundles were made for z5py-3.0.2-cp312-cp312-macosx_11_0_arm64.whl:

Publisher: wheels.yml on constantinpape/z5

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file z5py-3.0.2-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: z5py-3.0.2-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 891.5 kB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for z5py-3.0.2-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 62280d08c2b16a8e24ec616f31cc694aa657fa15785a39b96c7b99056739c2dd
MD5 0a6602ecfc762cef1ddea82c538a8192
BLAKE2b-256 a017877e69f8019c8669886724c9043255c901117d2aa75195db9b0e3660198b

See more details on using hashes here.

Provenance

The following attestation bundles were made for z5py-3.0.2-cp311-cp311-win_amd64.whl:

Publisher: wheels.yml on constantinpape/z5

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file z5py-3.0.2-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for z5py-3.0.2-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 3e0bd36b3d6b7d29d7fe7c146cc131651e95a7c60a4089bc1d0bbe79aa26045b
MD5 4ac6c5f8550ef240769c05ba8da437a8
BLAKE2b-256 bed7d98cf19928b5a887937092eab4b474056a306b5d92ce4e08cdacc9d247d7

See more details on using hashes here.

Provenance

The following attestation bundles were made for z5py-3.0.2-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl:

Publisher: wheels.yml on constantinpape/z5

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file z5py-3.0.2-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for z5py-3.0.2-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 434b83d197ad6ae94a43a41230c96bec8cbe384d2933e932277513263a37d6e3
MD5 42d222ab01d7ace5b13ea7e673bb035b
BLAKE2b-256 1df245e8b3d5864d0ca166f14e2c52eb40c2b8d1035d1a2f0ac6cb3603f1a3ee

See more details on using hashes here.

Provenance

The following attestation bundles were made for z5py-3.0.2-cp311-cp311-macosx_11_0_arm64.whl:

Publisher: wheels.yml on constantinpape/z5

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

3.0.2 This release

13 files

3.0.1

10 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page