Skip to main content

Tool for robustly creating and updating Zarr datacubes from smaller slices

Project description

 zappend

CI codecov PyPI Version Conda Version Code style: black Binder GitHub License


zappend is a tool written in Python that is used for robustly creating and updating Zarr datacubes from smaller dataset slices. It is built on top of the awesome Python packages xarray and zarr.

Motivation

The objective of zappend is enabling geodata scientists and developers to robustly create large data cubes. The tool performs transaction-based dataset appends to existing data cubes in the Zarr format. If an error occurs during an append step — typically due to I/O problems or out-of-memory conditions — zappend will automatically roll back the operation, ensuring that the existing data cube maintains its structural integrity. The design drivers behind zappend are first ease of use and secondly, high configurability regarding filesystems, data source types, data cube outline and encoding.

The tool comprises a command-line interface, a Python API for programmatic control, and a comprehensible documentation to guide users effectively. You can easily install zappend as a plain Python package using either pip install zappend or conda install -conda-forge zappend.

Features

The zappend tool provides the following features:

  • Locking: While the target dataset is being modified, a file lock is created, effectively preventing concurrent dataset modifications.
  • Transaction-based dataset appends: On failure during an append step, the transaction is rolled back, so that the target dataset remains valid and preserves its integrity.
  • Filesystem transparency: The target dataset may be generated and updated in any writable filesystems supported by the fsspec package. The same holds for the slice datasets to be appended.
  • Dataset polling: The tool can be configured to wait for slice datasets to become available.
  • Dynamic attributes: Use syntax {{ expression }} to update the target dataset with dynamically computed attribute values.
  • CLI and Python API: The tool can be used in a shell using the zappend command or from Python. When used from Python using the zappend() function, slice datasets can be passed as local file paths, URIs, as datasets of type xarray.Dataset, or as custom slice sources.

More about zappend can be found in its documentation.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

zappend-0.8.0.tar.gz (52.6 kB view details)

Uploaded Source

Built Distribution

zappend-0.8.0-py3-none-any.whl (52.7 kB view details)

Uploaded Python 3

File details

Details for the file zappend-0.8.0.tar.gz.

File metadata

  • Download URL: zappend-0.8.0.tar.gz
  • Upload date:
  • Size: 52.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.9.20

File hashes

Hashes for zappend-0.8.0.tar.gz
Algorithm Hash digest
SHA256 ec4f80fca8b36e1a442f007f1fd8d7fe792e1d741c267e595a5de95649a2fdd4
MD5 52199747fa47d5c11dfa50f9b5e5c2ba
BLAKE2b-256 f2a968eb6234381213005078c47109fb65dd02a37310228d7689c1ccc9f27429

See more details on using hashes here.

Provenance

File details

Details for the file zappend-0.8.0-py3-none-any.whl.

File metadata

  • Download URL: zappend-0.8.0-py3-none-any.whl
  • Upload date:
  • Size: 52.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.9.20

File hashes

Hashes for zappend-0.8.0-py3-none-any.whl
Algorithm Hash digest
SHA256 029181570968bae00c160842aab483defa7941813e0e2e3d9a08fdccc903d77f
MD5 26f05c9201c1ef39968b169e6d0bc67d
BLAKE2b-256 de0d236d3c6ecef57eee7edbafb21c9ae3e2df04dd99bd006a9be223566ab5ea

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page