zarr-copy 0.1.0

Copy a zarr dataset while changing chunking - variable by variable

zarr-copy

Copy a zarr dataset while optionally rechunking variables — variable by variable.

Installation

pip install -e .

Or with uv:

uv sync

Usage

zarr-copy [-h] [-v VAR [VAR ...]] [--rechunk DIM=SIZE [DIM=SIZE ...]]
          [--double-remap DIM] [--tmp-path DIR]
          zarr_in zarr_out

Positional arguments

Argument	Description
zarr_in	Path or S3 URI of the source zarr group
zarr_out	Path or S3 URI of the destination zarr group

Optional arguments

Flag	Description
-v VAR …	Variable names to copy. Copies all variables when omitted.
--rechunk DIM=SIZE …	Rechunk one or more dimensions, e.g. --rechunk time=1 lat=256.
--double-remap DIM	Use the two-pass copy strategy for the given dimension to reduce peak memory usage.
--tmp-path DIR	Directory for temporary files used by --double-remap. Defaults to the system temp dir.

Examples

Copy all variables without rechunking:

zarr-copy /data/input.zarr /data/output.zarr

Copy selected variables and rechunk the time dimension:

zarr-copy /data/input.zarr /data/output.zarr -v temperature salinity --rechunk time=1

Copy from S3, rechunk multiple dimensions, and use the double-remap strategy:

zarr-copy s3://my-bucket/input.zarr /data/output.zarr \
  --rechunk time=1 lat=256 lon=256 \
  --double-remap time \
  --tmp-path /scratch

Write to S3 from a local source:

zarr-copy /data/input.zarr s3://my-bucket/output.zarr \
  --rechunk lat=256 lon=256

Increase spatial chunk sizes for better read performance along lat/lon:

zarr-copy /data/input.zarr /data/output.zarr --rechunk lat=256 lon=256

Optimise for time-series access by making the time dimension unchunked (one chunk per step):

zarr-copy /data/input.zarr /data/output.zarr --rechunk time=1

Rechunk a large dataset where the time axis is being heavily reordered, using /scratch for intermediate storage to cap memory use:

zarr-copy /data/input.zarr /data/output.zarr \
  -v u v w temperature \
  --rechunk time=1 level=1 lat=180 lon=360 \
  --double-remap time \
  --tmp-path /scratch

Notes

• Rechunking requires _ARRAY_DIMENSIONS metadata on each array (standard for xarray/CF-convention zarr stores).
• S3 access is handled transparently via s3fs.
• The --double-remap strategy writes data to a temporary intermediate zarr (lz4-compressed) to avoid holding large in-memory buffers during rechunking.