zarrmony

Convert any bioimage file to OME-Zarr v0.5, preserving metadata.

Zarrmony reads proprietary microscopy formats (CZI, LIF, ND2, OME-TIFF, ...) via bioio and writes them as OME-Zarr v0.5, with mean-pool pyramid generation (--downsample-method max for sparse labels) and a full audit trail of the conversion. User-supplied metadata (study/treatment/etc.) is not handled by zarrmony — it is owned by aperture-backend, which associates OME-Zarr stores to a separate metadata database.

By default (--layout auto) the writer is chosen from the reader's layout_hint: a flat reader writes one self-describing <scene>.ome.zarr store per scene under the output directory; a plate-shaped reader writes a single OME-NGFF HCS plate store at the output. The legacy bundled bioformats2raw.layout shape is opt-in via --layout bf2raw (CLI) or layout="bf2raw" (library).

Status: v0.15 in active development. API and metadata schema are not yet stable.

Install

pip install zarrmony

Readers

Zarrmony dispatches to a reader plugin per input format. They come in three tiers:

• Built-in (bundled by default): CZI, LIF, ND2.
• Optional extras in this repo (opt-in via pip install "zarrmony[<extra>]"): OME-TIFF via the ome-tiff extra, and ~150 vendor formats via the bioformats extra (GPL-3.0 — see below).
• External plugins (separate PyPI distributions, entry-point registered):
  • zarrmony-phenix — Opera Phenix (wraps pyphenix.OperaPhenixReader) — pip install zarrmony-phenix
  • zarrmony-blaze — Miltenyi UltraMicroscope Blaze (MACS iQ-processed) — pip install zarrmony-blaze
  • zarrmony-snouty — Snouty single-objective light-sheet — pip install zarrmony-snouty
  • zarrmony-smartspim — LifeCanvas SmartSPIM stitched exports — pip install zarrmony-smartspim

Extras

Bio-Formats-backed vendor formats

pip install "zarrmony[bioformats]"

What it buys. Everything on the Bio-Formats supported-formats list that no permissively-licensed bioio backend covers — around 150 formats. The motivating case is Olympus/Evident cellSens VSI whole-slide data; Zeiss ZVI and Hamamatsu NDPI are in the same bucket. No zarrmony code is involved: once bioio-bioformats is installed, the built-in bioio catch-all plugin dispatches to it, and the audit record's distribution field names it. Point zarrmony at the .vsi file itself — Bio-Formats follows the .ets sidecar directory automatically. The audit records that it did: input.size_bytes is the .vsi alone (a few MB of index), and input.files is the whole set Bio-Formats read, with input.size_is_partial saying which is which. --checksum covers both — the named path under input.sha256, the whole set under input.files.sha256.

Licence. bioio-bioformats is GPL-3.0; Bio-Formats is GPL. Installing this extra puts GPL code in your environment. Zarrmony itself remains Apache-2.0 and no GPL package is in its default dependency closure — which is exactly why this extra is opt-in and why it is excluded from all. Do not add it there. See ADR-0011.

Java. Bio-Formats needs a JVM, but not one you have to install: bffile / scyjava / cjdk fetch their own JDK (~36 MiB, once) and the Bio-Formats maven artifacts on first use. No system Java, no maven, no JAVA_HOME. The first file you open is slow while that downloads; everything after is cached.

Gigapixel inputs need tiling. bioio-bioformats returns one dask chunk per plane by default. On a 141k × 168k slide that is a single 47.5 GB chunk, and the writer will try to hold it in memory to rechunk it. Pass:

zarrmony convert slide.vsi out/ --reader-kwarg dask_tiles=true

Leave tile_size off. Zarrmony plans the output geometry first and then asks the reader for tiles that fit it exactly, recording the choice in config.reader_tile_size. Pinning your own is supported and sometimes right, but a tile that does not divide the write grid makes every write split a source tile — on the reference slide that is 831,936 dask tasks against 369,600 — so the writer warns and names the tile that would have worked.

Usage

CLI

# Auto (default): dispatches on the reader's layout_hint.
#   flat readers (CZI, LIF, ND2, OME-TIFF) → per-scene stores under OUTPUT
#   plate-shaped readers (e.g. zarrmony-phenix) → a single HCS plate store at OUTPUT
zarrmony convert input.czi output_dir/

# Force per-scene (one <scene>.ome.zarr store per scene under OUTPUT).
zarrmony convert input.czi output_dir/ --layout per-scene

# Force HCS plate (one <plate>.ome.zarr store at OUTPUT). Requires a
# plate-shaped reader; flat readers raise LayoutMismatchError.
zarrmony convert phenix-acquisition/ output.ome.zarr --layout plate

# Bundled bioformats2raw.layout (opt-in): writes a single store at OUTPUT.
zarrmony convert input.czi output.ome.zarr --layout bf2raw

# LIF-specific: write one OME-Zarr per mosaic tile (with stage positions in
# <Plane>) instead of bioio-lif's auto-stitched 1-pixel-overlap output.
# See docs/adr/0005-lif-mosaic-write-strategy.md.
zarrmony convert mosaic.lif output_dir/ --lif-mosaic per-tile

zarrmony inspect input.czi

Library

from zarrmony import convert

# Auto (default): for a flat reader, returns {"input": ..., "stores": [...]};
# for a plate-shaped reader, returns the single plate audit dict (schema 3,
# with "fields" and a top-level "plate" block). Switch on audit["layout"].
result = convert("input.lif", "output_dir/")

# Bundled: returns the single bundle's audit dict.
audit = convert("input.lif", "output.ome.zarr", layout="bf2raw")

# HCS plate: writes one OME-NGFF plate store at OUTPUT.
audit = convert("phenix-acquisition/", "output.ome.zarr", layout="plate")

Output geometry

Zarrmony plans an output store's geometry — how many pyramid levels there are, what each level's extent is, and how each level is divided into chunks — itself, rather than delegating chunk shape to bioio-ome-zarr's memory-target heuristic. Every constant below traces to a measurement; ADR-0010 records which, and what was rejected.

What the planner does by default:

• Chunks are world-cubic and small. Each level gets the largest power-of-two chunk whose raw size fits the 512 KiB target and whose extents are closest to cubic in micrometres, planned against that level's own voxel spacing. Near-isotropic uint16 data lands on the familiar 1,1,64,64,64; a 10:1 confocal stack (Z 5 µm, XY 0.5 µm) lands on 1,1,16,128,128 — 80 × 64 × 64 µm — instead of a voxel-cubic 64³ spanning 320 × 32 × 32 µm. T and C are never chunked, so a viewer fetching one channel at one timepoint never pays for the others.
• The pyramid moves toward isotropy. A level halves every spatial axis whose physical spacing is within isotropy_tolerance of the finest still-halvable axis's, so the scarce axis — Z, for most volumetric light microscopy — is spent last. No axis halves below axis_floor voxels, and an axis already below it never halves: a 3-plane stack keeps its 3 planes at every level.
• Depth is the greater of two rules. The pyramid_min_size Y/X floor, and the depth at which a level becomes a coarse level — one a viewer can decode whole and use as spatial context, meaning Z·Y·X·itemsize per timepoint and channel is at most coarse_max_bytes and the longest lateral axis is at most coarse_max_long_axis. Because depth is a max(), no conversion loses a level. A pyramid that bottoms out at the axis floor while still too large simply has no coarse level.
• Levels above 0 are mean-pooled, uniformly. downsample_method="max" switches the whole pyramid to max-pool for sparse-label acquisitions, where mean-pooling dissolves small objects into the background.
• The same rules apply to per-scene, bf2raw and plate output, with no Z > 1 gate and no 2D exemption: every rule is written over the axes that are present, so a 2160² plate field is planned by exactly the rule a whole-brain volume is.

Knobs

Every field lives on the frozen zarrmony.Geometry policy object, passed as convert(..., geometry=...). Most are also CLI flags on zarrmony convert.

The two coarse-level bounds are the defaults of the viewer this output is tuned for; they are fields rather than constants so a store can be planned for a consumer with a different budget.

# Fewer objects, same read granularity: 512 KiB chunks packed into 8 MiB
# shards. Only for consumers that read sharded zarr v3 — see below.
zarrmony convert slide.vsi output_dir/ --shard-target-bytes

# Bigger chunks: fewer objects, coarser culling. For object storage where
# listing cost matters more than round-trip latency.
zarrmony convert input.czi output_dir/ --chunk-target-bytes 2097152

# Sparse labels: keep peak intensity in the pyramid.
zarrmony convert labels.czi output_dir/ --downsample-method max

from zarrmony import Geometry, convert

audit = convert(
    "input.czi",
    "output_dir/",
    geometry=Geometry(chunk_target_bytes=2 * 1024 * 1024, downsample_method="max"),
)

chunk_shape and pyramid_min_size are also retained directly on convert() as sugar that folds into a default policy, so callers written before the policy object keep working. Passing geometry= together with either raises ValueError rather than silently picking a winner.

The resolved policy is recorded in the audit under config.geometry, and what it produced is recorded per scene / per field as level_shapes, chunk_shapes, shard_shapes and coarse_level_index — so "does this store have a level a viewer can hold whole?" is answerable from the store's own metadata.

Object count, and sharding

Small chunks trade bytes-per-object for objects. A whole-brain light-sheet store goes from 87,048 objects to ~3.2 M (~37×); a 2160² plate field goes from 4 to 39; a gigapixel slide scene reaches ~370k objects at level 0 alone. On local disk that is irrelevant. On GCS/S3 it is listing time plus per-object metadata cost, and at slide scale it is also conversion wall-clock: one such scene sustains 55 chunks/min with perfectly aligned reader tiles, which is about six days for its pyramid, against 3 h 02 m for the same scene written in 8 MiB units. Each object is 16× smaller but takes 2.8× longer to write, so the cost is graph size rather than bytes — which is why sharding fixes it and a bigger chunk only trades it for a worse viewer.

You do not have to work this out per scene. The planner knows every level's grid before it writes anything, so it knows the count: an unsharded scene whose pyramid plans more than 100,000 objects emits an ObjectCountWarning at plan time naming the count, --shard-target-bytes, the count sharding would give instead, and the reader catch below. Nothing is refused — the run is correct and the count may be what you want — and the ordinary warnings filters silence it.

Sharding answers this without giving up read granularity, because the shard is the write unit and the chunk is the read unit. --shard-target-bytes packs whole chunks into 8 MiB storage objects: that slide scene's level 0 becomes 512² chunks inside 2048² shards, 369,600 objects down to 23,184, with each 512 KiB chunk still individually range-readable. Shards are planned by the same world-cubic rule as chunks, per level, so an isotropic volume at the defaults gets a 128 × 128 × 256 shard holding 16 chunks of 64³ — 3.2 M objects down to 210,345 on the whole-brain store.

Those object counts are the planner's arithmetic. They are exact on the slide and an upper bound on the volume, since zarr skips a shard whose chunks are all fill value: the whole-brain store landed at 209,211 shards, 1,134 under its grid, all of them an empty sliver of a padded axis. The wall-clock is now measured too. That same whole-slide scene, re-converted at 512²-in-2048²:

So sharding costs 1.6 % of wall-clock and buys a read unit 16× smaller. The object count landed on the planner's prediction exactly at all ten levels, and the +19 % CPU is compression overhead — 493,484 blocks of 512 KiB instead of 31,156 of 8 MiB — which parallelises rather than showing up as elapsed time.

Earlier text here warned that the store would be larger, since small chunks compress worse. It is not, and the reason is worth stating because it also says the warning was aimed at the wrong thing. Compression runs per chunk, and sharding does not change the chunk — it only packs chunks that already exist into one object. So the compressed payload of a sharded store is the payload of the unsharded one, and the only cost is the shard index: 16 bytes per chunk, plus 4 bytes per shard.

Both halves are measured. Shrinking the chunk 16× on the slide scene above cost nothing — 88 MB on a 139 GiB store, and in the cheaper direction. And the whole-brain volume has now been written both ways at the same 64³ chunk, which isolates sharding by itself:

The 51.4 MB difference is the index and nothing else: 3,194,997 chunk slots × 16 B plus 209,211 shards × 4 B predicts 51.96 MB, within 1.2 % of what landed on disk. Full run details in #124 (slide) and #126 (volume).

It is off by default, because it changes who can read the store. Chunks stay individually readable and every zarr-python 3 consumer is unaffected — napari-ome-zarr, dask, plain __getitem__, subsets straddling either grid, all verified byte-identical against an unsharded store. But a consumer that parses the codec chain itself sees sharding_indexed where it expects bytes and refuses the store: lucida-store accepts only [bytes] or [bytes, compressor], so a sharded store fails there with first storage codec must be 'bytes', got 'sharding_indexed'. The CLI warns whenever sharding is on. See ADR-0010 for the measurements and the reversal.

Extending zarrmony

Add support for a new bioimage format by writing a reader plugin. See Writing a zarrmony reader plugin for the Reader Protocol, matcher conventions, entry-point registration, and a worked example.

License

Apache-2.0. See LICENSE.