Skip to main content

Checksum support for zarrs stored in various backends

Project description

zarr_checksum

Algorithms for calculating a zarr checksum against local or cloud storage

Install

pip install zarr-checksum

Usage

CLI

To calculate the checksum for a local zarr archive

zarrsum local <directory>

To calcuate the checksum for a remote (S3) zarr archive

zarrsum remote s3://your_bucket/prefix_to_zarr

Python

To calculate the checksum for a local zarr archive

from zarr_checksum import compute_zarr_checksum
from zarr_checksum.generators import yield_files_local, yield_files_s3

# Local
checksum = compute_zarr_checksum(yield_files_local("local_path"))

# Remote
checksum = compute_zarr_checksum(
    yield_files_s3(
        bucket="your_bucket",
        prefix="prefix_to_zarr",
        # Credentials can also be passed via environment variables
        credentials={
            aws_access_key_id: "youraccesskey",
            aws_secret_access_key: "yoursecretkey",
            region_name: "us-east-1",
        }
    )
)

Access checksum information

>>> checksum.digest
'c228464f432c4376f0de6ddaea32650c-37481--38757151179'
>>> checksum.md5
'c228464f432c4376f0de6ddaea32650c'
>>> checksum.count
37481
>>> checksum.size
38757151179

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

zarr_checksum-0.2.3.tar.gz (7.9 kB view hashes)

Uploaded Source

Built Distribution

zarr_checksum-0.2.3-py3-none-any.whl (9.2 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page