Tool for robustly creating and updating Zarr datacubes from smaller slices
Project description
zappend
zappend
is a tool written in Python that is used for robustly creating and
updating Zarr datacubes from smaller dataset slices. It is built on top of the
awesome Python packages xarray and zarr.
Motivation
The objective of zappend
is enabling geodata scientists and developers to
robustly create large data cubes. The tool performs transaction-based dataset
appends to existing data cubes in the
Zarr format. If an error
occurs during an append step — typically due to I/O problems or out-of-memory
conditions — zappend
will automatically roll back the operation, ensuring that
the existing data cube maintains its structural integrity. The design drivers
behind zappend are first ease of use and secondly, high configurability
regarding filesystems, data source types, data cube outline and encoding.
The tool comprises a command-line interface, a Python API for programmatic
control, and a comprehensible documentation to guide users effectively.
You can easily install zappend
as a plain Python package using either
pip install zappend
or conda install -conda-forge zappend
.
Features
The zappend
tool provides the following features:
- Locking: While the target dataset is being modified, a file lock is created, effectively preventing concurrent dataset modifications.
- Transaction-based dataset appends: On failure during an append step, the transaction is rolled back, so that the target dataset remains valid and preserves its integrity.
- Filesystem transparency: The target dataset may be generated and updated in any writable filesystems supported by the fsspec package. The same holds for the slice datasets to be appended.
- Dataset polling: The tool can be configured to wait for slice datasets to become available.
- Dynamic attributes: Use syntax
{{ expression }}
to update the target dataset with dynamically computed attribute values. - CLI and Python API: The tool can be used in a shell using the
zappend
command or from Python. When used from Python using thezappend()
function, slice datasets can be passed as local file paths, URIs, as datasets of type xarray.Dataset, or as custom slice sources.
More about zappend can be found in its documentation.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file zappend-0.8.0.tar.gz
.
File metadata
- Download URL: zappend-0.8.0.tar.gz
- Upload date:
- Size: 52.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.9.20
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | ec4f80fca8b36e1a442f007f1fd8d7fe792e1d741c267e595a5de95649a2fdd4 |
|
MD5 | 52199747fa47d5c11dfa50f9b5e5c2ba |
|
BLAKE2b-256 | f2a968eb6234381213005078c47109fb65dd02a37310228d7689c1ccc9f27429 |
File details
Details for the file zappend-0.8.0-py3-none-any.whl
.
File metadata
- Download URL: zappend-0.8.0-py3-none-any.whl
- Upload date:
- Size: 52.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.9.20
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 029181570968bae00c160842aab483defa7941813e0e2e3d9a08fdccc903d77f |
|
MD5 | 26f05c9201c1ef39968b169e6d0bc67d |
|
BLAKE2b-256 | de0d236d3c6ecef57eee7edbafb21c9ae3e2df04dd99bd006a9be223566ab5ea |