Performant streaming to Zarr storage, on filesystem or cloud

Project description

Acquire Zarr streaming library

This library supports chunked, compressed, multiscale streaming to Zarr, with OME-NGFF metadata.

This code builds targets for python and C.

For python: pip install acquire-zarr

Building

Installing dependencies

This library has the following dependencies:

c-blosc v1.21.5
nlohmann-json v3.11.3
minio-cpp v0.3.0
crc32c v1.1.2

We use vcpkg to install them, as it integrates well with CMake. To install vcpkg, clone the repository and bootstrap it:

git clone https://github.com/microsoft/vcpkg.git
cd vcpkg && ./bootstrap-vcpkg.sh

and then add the vcpkg directory to your path. If you are using bash, you can do this by running the following snippet from the vcpkg/ directory:

cat >> ~/.bashrc <<EOF
export VCPKG_ROOT=${PWD}
export PATH=\$VCPKG_ROOT:\$PATH
EOF

If you're using Windows, learn how to set environment variables here. You will need to set both the VCPKG_ROOT and PATH variables in the system control panel.

On the Mac, you will also need to install OpenMP using Homebrew:

brew install libomp

Configuring

To build the library, you can use CMake:

cmake --preset=default -B /path/to/build /path/to/source

On Windows, you'll need to specify the target triplet to ensure that all dependencies are built as static libraries:

cmake --preset=default -B /path/to/build -DVCPKG_TARGET_TRIPLET=x64-windows-static /path/to/source

Aside from the usual CMake options, you can choose to disable tests by setting BUILD_TESTING to OFF:

cmake --preset=default -B /path/to/build -DBUILD_TESTING=OFF /path/to/source

To build the Python bindings, make sure pybind11 is installed. Then, you can set BUILD_PYTHON to ON:

cmake --preset=default -B /path/to/build -DBUILD_PYTHON=ON /path/to/source

Building

After configuring, you can build the library:

cmake --build /path/to/build

Installing for Python

To install the Python bindings, you can run:

pip install .

[!NOTE] It is highly recommended to use virtual environments for Python, e.g. using venv or conda. In this case, make sure pybind11 is installed in this environment, and that the environment is activated before installing the bindings.

Usage

The library provides two main interfaces. First, ZarrStream, representing an output stream to a Zarr dataset. Second, ZarrStreamSettings to configure a Zarr stream.

A typical use case for a 4-dimensional acquisition might look like this:

ZarrStreamSettings settings = (ZarrStreamSettings){
    .store_path = "my_stream.zarr",
    .data_type = ZarrDataType_uint16,
    .version = ZarrVersion_3,
};
settings.store_path = "my_stream.zarr";
settings.data_type = ZarrDataType_uint16;
settings.version = ZarrVersion_3;

ZarrStreamSettings_create_dimension_array(&settings, 4);
settings.dimensions[0] = (ZarrDimensionProperties){
    .name = "t",
    .type = ZarrDimensionType_Time,
    .array_size_px = 0,      // this is the append dimension
    .chunk_size_px = 100,    // 100 time points per chunk
    .shard_size_chunks = 10, // 10 chunks per shard
};

settings.dimensions[1] = (ZarrDimensionProperties){
    .name = "c",
    .type = ZarrDimensionType_Channel,
    .array_size_px = 3,     // 3 channels
    .chunk_size_px = 1,     // 1 channel per chunk
    .shard_size_chunks = 1, // 1 chunk per shard
};

settings.dimensions[2] = (ZarrDimensionProperties){
    .name = "y",
    .type = ZarrDimensionType_Space,
    .array_size_px = 1080,  // height
    .chunk_size_px = 270,   // 4 x 4 tiles of size 270 x 480
    .shard_size_chunks = 2, // 2 x 2 tiles per shard
};

settings.dimensions[3] = (ZarrDimensionProperties){
    .name = "x",
    .type = ZarrDimensionType_Space,
    .array_size_px = 1920,  // width
    .chunk_size_px = 480,   // 4 x 4 tiles of size 270 x 480
    .shard_size_chunks = 2, // 2 x 2 tiles per shard
};

ZarrStream* stream = ZarrStream_create(&settings);

size_t bytes_written;
ZarrStream_append(stream, my_frame_data, my_frame_size, &bytes_written);
assert(bytes_written == my_frame_size);

Look at acquire.zarr.h for more details.

This acquisition in Python would look like this:

import acquire_zarr as aqz
import numpy as np

settings = aqz.StreamSettings(
    store_path="my_stream.zarr",
    data_type=aqz.DataType.UINT16,
    version=aqz.ZarrVersion.V3
)

settings.dimensions.extend([
    aqz.Dimension(
        name="t",
        type=aqz.DimensionType.TIME,
        array_size_px=0,
        chunk_size_px=100,
        shard_size_chunks=10
    ),
    aqz.Dimension(
        name="c",
        type=aqz.DimensionType.CHANNEL,
        array_size_px=3,
        chunk_size_px=1,
        shard_size_chunks=1
    ),
    aqz.Dimension(
        name="y",
        type=aqz.DimensionType.SPACE,
        array_size_px=1080,
        chunk_size_px=270,
        shard_size_chunks=2
    ),
    aqz.Dimension(
        name="x",
        type=aqz.DimensionType.SPACE,
        array_size_px=1920,
        chunk_size_px=480,
        shard_size_chunks=2
    )
])

# Generate some random data: one time point, all channels, full frame
my_frame_data = np.random.randint(0, 2**16, (3, 1080, 1920), dtype=np.uint16)

stream = aqz.ZarrStream(settings)
stream.append(my_frame_data)

S3

The library supports writing directly to S3-compatible storage. Configuration requires specifying the endpoint, bucket name, and region:

ZarrStreamSettings settings = { /* ... */ };

// Configure S3 storage
ZarrS3Settings s3_settings = {
    .endpoint = "https://s3.amazonaws.com",
    .bucket_name = "my-zarr-data",
    .region = "us-east-1"
};

settings.s3_settings = &s3_settings;

In Python, S3 configuration looks like:

import acquire_zarr as aqz

settings = aqz.StreamSettings()
# ...

# Configure S3 storage
s3_settings = aqz.S3Settings(
    endpoint="s3.amazonaws.com",
    bucket_name="my-zarr-data",
    region="us-east-1"
)

# Apply S3 settings to your stream configuration
settings.s3 = s3_settings

The library authenticates with S3 exclusively through environment variables:

AWS_ACCESS_KEY_ID: Your AWS access key
AWS_SECRET_ACCESS_KEY: Your AWS secret key

These must be set in the environment where your application runs.

Project details

Release history Release notifications | RSS feed

This version

0.4.0

Apr 24, 2025

0.3.1

Apr 22, 2025

0.3.0

Apr 18, 2025

0.2.4

Mar 25, 2025

0.2.3

Mar 12, 2025

0.2.2

Feb 25, 2025

0.2.1

Feb 25, 2025

0.2.0

Feb 11, 2025

0.1.0

Jan 21, 2025

0.0.5

Jan 9, 2025

0.0.2

Nov 26, 2024

0.0.1

Nov 8, 2024

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

acquire_zarr-0.4.0.tar.gz (69.8 kB view details)

Uploaded Apr 24, 2025 Source

Built Distributions

acquire_zarr-0.4.0-cp313-cp313-win_amd64.whl (2.6 MB view details)

Uploaded Apr 24, 2025 CPython 3.13Windows x86-64

acquire_zarr-0.4.0-cp313-cp313-manylinux_2_35_x86_64.whl (4.5 MB view details)

Uploaded Apr 24, 2025 CPython 3.13manylinux: glibc 2.35+ x86-64

acquire_zarr-0.4.0-cp313-cp313-macosx_14_0_universal2.whl (3.5 MB view details)

Uploaded Apr 24, 2025 CPython 3.13macOS 14.0+ universal2 (ARM64, x86-64)

acquire_zarr-0.4.0-cp313-cp313-macosx_13_0_universal2.whl (3.3 MB view details)

Uploaded Apr 24, 2025 CPython 3.13macOS 13.0+ universal2 (ARM64, x86-64)

acquire_zarr-0.4.0-cp312-cp312-win_amd64.whl (2.6 MB view details)

Uploaded Apr 24, 2025 CPython 3.12Windows x86-64

acquire_zarr-0.4.0-cp312-cp312-manylinux_2_35_x86_64.whl (4.5 MB view details)

Uploaded Apr 24, 2025 CPython 3.12manylinux: glibc 2.35+ x86-64

acquire_zarr-0.4.0-cp312-cp312-macosx_14_0_universal2.whl (3.5 MB view details)

Uploaded Apr 24, 2025 CPython 3.12macOS 14.0+ universal2 (ARM64, x86-64)

acquire_zarr-0.4.0-cp312-cp312-macosx_13_0_universal2.whl (3.3 MB view details)

Uploaded Apr 24, 2025 CPython 3.12macOS 13.0+ universal2 (ARM64, x86-64)

acquire_zarr-0.4.0-cp311-cp311-win_amd64.whl (2.6 MB view details)

Uploaded Apr 24, 2025 CPython 3.11Windows x86-64

acquire_zarr-0.4.0-cp311-cp311-manylinux_2_35_x86_64.whl (4.5 MB view details)

Uploaded Apr 24, 2025 CPython 3.11manylinux: glibc 2.35+ x86-64

acquire_zarr-0.4.0-cp311-cp311-macosx_14_0_universal2.whl (3.5 MB view details)

Uploaded Apr 24, 2025 CPython 3.11macOS 14.0+ universal2 (ARM64, x86-64)

acquire_zarr-0.4.0-cp311-cp311-macosx_13_0_universal2.whl (3.3 MB view details)

Uploaded Apr 24, 2025 CPython 3.11macOS 13.0+ universal2 (ARM64, x86-64)

acquire_zarr-0.4.0-cp310-cp310-win_amd64.whl (2.6 MB view details)

Uploaded Apr 24, 2025 CPython 3.10Windows x86-64

acquire_zarr-0.4.0-cp310-cp310-manylinux_2_35_x86_64.whl (4.5 MB view details)

Uploaded Apr 24, 2025 CPython 3.10manylinux: glibc 2.35+ x86-64

acquire_zarr-0.4.0-cp310-cp310-macosx_14_0_universal2.whl (3.5 MB view details)

Uploaded Apr 24, 2025 CPython 3.10macOS 14.0+ universal2 (ARM64, x86-64)

acquire_zarr-0.4.0-cp310-cp310-macosx_13_0_x86_64.whl (3.3 MB view details)

Uploaded Apr 24, 2025 CPython 3.10macOS 13.0+ x86-64

acquire_zarr-0.4.0-cp39-cp39-win_amd64.whl (2.6 MB view details)

Uploaded Apr 24, 2025 CPython 3.9Windows x86-64

acquire_zarr-0.4.0-cp39-cp39-manylinux_2_35_x86_64.whl (4.5 MB view details)

Uploaded Apr 24, 2025 CPython 3.9manylinux: glibc 2.35+ x86-64

acquire_zarr-0.4.0-cp39-cp39-macosx_14_0_universal2.whl (3.5 MB view details)

Uploaded Apr 24, 2025 CPython 3.9macOS 14.0+ universal2 (ARM64, x86-64)

acquire_zarr-0.4.0-cp39-cp39-macosx_13_0_x86_64.whl (3.3 MB view details)

Uploaded Apr 24, 2025 CPython 3.9macOS 13.0+ x86-64

File details

Details for the file acquire_zarr-0.4.0.tar.gz.

File metadata

Download URL: acquire_zarr-0.4.0.tar.gz
Upload date: Apr 24, 2025
Size: 69.8 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for acquire_zarr-0.4.0.tar.gz
Algorithm	Hash digest
SHA256	`8ecd0707bda65b629e2efd4d525d4ee775cfa723bf8414806a5b8b6dad307b60`
MD5	`2cc30956d9eaf44dca62469e6ad0d468`
BLAKE2b-256	`185a54133b3af4f1f2866c91587caeaa862b2b223b66e63c5f102b2bc5a51b56`

Algorithm	Hash digest
SHA256	`65b14ed93a42fc43f72236b3b8cb3977910d16ef19fc6cfaa3d567a727be7190`
MD5	`8d26aaff3c56d2fdfde5a81cbef8c34d`
BLAKE2b-256	`bf589f0fc821d578ae356c75c6274ad42c5863061dc7e5a8d54fcd8b23ab4770`

Algorithm	Hash digest
SHA256	`a2c65a4fc53252d6b1b4b6081e72cab9167c590241e1de726ff4cad5bc908ffd`
MD5	`49ed91f3f6d4c78db6d000c5d35fffe8`
BLAKE2b-256	`16cedb2a5a81c32fba09d5e2de32b7e0f694dd49536fd4f11071b57ca8e56461`

Algorithm	Hash digest
SHA256	`d3ce1635a9654ffe09b59f6f47899986b1d0dad1f4df6aed5b419f7876643eb9`
MD5	`dcd4407bc65b75d7a7e9ac1275ff06d8`
BLAKE2b-256	`cd02cc7e2f96251cc06924f409ad32f7c4393a88c3801d0fa07d2a95a64338a7`

Algorithm	Hash digest
SHA256	`09607e9abe5403879b3180e47d82918ecfa481f110baf7dc2ce586ae40cadfe2`
MD5	`ede6ef7a7134a76c85de56ea9900cce6`
BLAKE2b-256	`7f17cdec8a8c309b015dea9f258dd6b49b436ea477d370bbf5dca0c30f67d875`

Algorithm	Hash digest
SHA256	`42a13d1234a423a446ae776fa7a1c43b0439542af38617b9164ac1dd36ae1312`
MD5	`478473790987656c3e2c768b3b843028`
BLAKE2b-256	`61758be6095139ae3821f5b636d89379744481d182312681367f3ef78b05e92c`

Algorithm	Hash digest
SHA256	`fb97a998693518e2fc648336e4e3579a48eaaad45f98873cbe705adbfe3c4cb2`
MD5	`fc60fb0c85b0a71290164e5c49f6bc1d`
BLAKE2b-256	`3d2906237b735fb1ecf7990a0b4734b9383a68e729d3fef8fa098ab31a98ea3a`

Algorithm	Hash digest
SHA256	`0bda707adbdf0b3def55d1080cab1ee63fbe5f82f12c6da6830eb8ad92914a6c`
MD5	`86bd68b2f8a36efdf91b7080804b81c4`
BLAKE2b-256	`ef4d57d3f97e9245c22a6d274ea7da35660132ee0c0d0189dde14851ddee7a41`

Algorithm	Hash digest
SHA256	`00b53a5c7c8485ef3459f3f43f978f047d07c92c9b1c7b89cc8bf6e1eee1df10`
MD5	`7625626f0707fa038a2ad117742a5ad9`
BLAKE2b-256	`7a65cc882e00462ef0573ace8a3bf523e2f353dc72c5c02e0fbf889436ac934f`

Algorithm	Hash digest
SHA256	`ff441ccb40694a7b2e9f725c3e5a47797e983b6b60d30a6f5d06c1044bb07d5f`
MD5	`474ace2e4b43cdc09eb6d09e45c3753b`
BLAKE2b-256	`a6880cb797b6360388fa364d169ba1c46dc505858f1b68deee998ad3174b63b5`

Algorithm	Hash digest
SHA256	`87be846a2e9ec953374dfc01026fed7d41b902823f8429775f3c5e5c6bb2beb2`
MD5	`26ce1338240ffb0217f4b0c3a14352b6`
BLAKE2b-256	`b130aafeb889973bc4e3c34bfba4765b4087f77ce321321a5483d2d5a83f0b78`

Algorithm	Hash digest
SHA256	`d5f04f2e4beef5fd79cb4e508ca8f8d3c6cf06e4e239e2d9cf8046edb996b355`
MD5	`1ddc968912267ca304c64d9bed995b98`
BLAKE2b-256	`cb0eddd6438936b92693b2002436f79326068bd0125f00f58709516a215c712c`

Algorithm	Hash digest
SHA256	`c5d1391976af7fe5571a075e3c58f5f17fa836f46b7044a883fae9665c1fcadb`
MD5	`b86d421cc30d9fca84e340f5934e4e54`
BLAKE2b-256	`35fad67b2d0c37005c5fdb1073d81d7f0d1418a2426bbeb512c1be86ac72a2f4`

Algorithm	Hash digest
SHA256	`a7863b06f69c274f6c43bc65a727d8054a6e3196d06346a6aa89d1a54499794d`
MD5	`1279c7d03541335642735ce6701663eb`
BLAKE2b-256	`a07936aceba139f014425a34181c0aa55b8dd0972afc4d1ead87f3539e916f4a`

Algorithm	Hash digest
SHA256	`36ac2f7981fe96ec115d70eafc88abe4f7ee4450d06f4f926ea1b672e78776a1`
MD5	`28256d9177028334b2b779610b9927a4`
BLAKE2b-256	`6ac113b1f2540600933f1a0149f6333c998edd3bc931284c5b368290573ab20b`

Algorithm	Hash digest
SHA256	`4d2402c621bbeb091b9e97946815fd4722f38b27a1205c84268788b0c794a3bb`
MD5	`8592891512b5b6cb1d948c12cf793ea1`
BLAKE2b-256	`ae178a4e374b1751ba63c2d065238ece781836c3af24f1afbcee7158bc9a8a50`

Algorithm	Hash digest
SHA256	`400eb4ab28d8570b83df49d92f1d86b3b26748b8e834615f9210c67178b424d8`
MD5	`08c52aecb89fd4b438b5c948f2811574`
BLAKE2b-256	`695e1909e28ac838684ad3dd783b80530b675a8192d86946fa146f21c4893293`

Algorithm	Hash digest
SHA256	`a0d096ba592d664ac321322d71a8093137ce17920b34c46e54de89710c74db65`
MD5	`0b70eeec6a65fa5bda8d30138bfb1d48`
BLAKE2b-256	`1946f11c0a6ff4e94dcd3de9e94d0f7dbab26d299ccfdbf3d427226408cca470`

Algorithm	Hash digest
SHA256	`efb07e34d5ab1010f2ae219864dec588cf61eeedb3144c52474a04379ce74631`
MD5	`beb0235debaa965a3524f3f2b4972932`
BLAKE2b-256	`4485b957ce9a0772d17eab79eb1ad9c813e338bde89e66a019b15dd946c9c123`

Algorithm	Hash digest
SHA256	`1a79ca05b9f1227df6350c3ae57b19ca0ecd5310cdb8095f311f3afcaa11ea9f`
MD5	`5a48cb1d80964aec365a199cc4ce68fe`
BLAKE2b-256	`e3e25f617e6a147a5458e0e3031159153d359c298a96eb9f999d8cde7c2bc4a1`

Algorithm	Hash digest
SHA256	`b8d6f2b21fb0b83641f9ce524be0e477fccc7fb1aa2eb3257e5b6016f191302e`
MD5	`ac5bc51fe4c0d6314198912fe91da898`
BLAKE2b-256	`4ca793f318a9024f70717002db662ea1bd3ab567aa365e1b89354feb5cb0de7e`

acquire-zarr 0.4.0

Navigation

Verified details

Owner

Maintainers

Unverified details

Meta

Project description

Acquire Zarr streaming library

Building

Installing dependencies

Configuring

Building

Installing for Python

Usage

S3

Project details

Verified details

Owner

Maintainers

Unverified details

Meta

Release history Release notifications | RSS feed

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata