genome-zarr-4bit

genome-zarr-4bit turns a FASTA file into a standard, directory-backed Zarr v3 group. Each chromosome is a one-dimensional uint8 Zarr array containing two 4-bit base codes per byte: A=0, C=1, G=2, T=3, and N (or any unsupported IUPAC base) =4. This is the same packed layout used by zarr_compression_benchmark.py; unlike its manual backend, Zarr owns the chunking and compression.

The implementation scans the FASTA once for record sizes and streams it on the second pass. It does not load chromosomes into memory. The default logical chunk is 1 MiB of bases (512 KiB packed bytes), matching the benchmark's default 1 MB base chunk.

Install

From this directory:

python -m pip install .

The package requires Zarr v3 and explicitly creates every group with zarr_format=3. Zstandard compression uses Zarr v3's native zarr.codecs.ZstdCodec; no numcodecs dependency is used for this codec. Transcode commands can read an existing package-created v2 store as a migration source, but every destination they create is Zarr v3.

Commands

# FASTA -> packed 4-bit Zarr with Zstandard compression
genome-zarr fasta-to-zstd genome.fa /data/genome.zarr

# compressed -> uncompressed, preserving arrays, chunks, and attributes
genome-zarr decompress /data/genome.zarr /data/genome-uncompressed.zarr

# uncompressed -> Zstandard (also works as a recompression operation)
genome-zarr compress /data/genome-uncompressed.zarr /data/genome-recompressed.zarr

Destinations must be new unless --overwrite is supplied. To select a chunk size or Zstandard level or number of CPU workers:

# FASTA -> packed 4-bit Zarr with Zstandard compression (chunk size, level, workers)
genome-zarr fasta-to-zstd genome.fa /data/genome.zarr --chunk-bases 2097152 --zstd-level 6 --num-workers 16

# compressed -> uncompressed with parallel workers
genome-zarr decompress /data/genome.zarr /data/genome-uncompressed.zarr --num-workers 8

# uncompressed -> Zstandard (recompression) with parallel workers
genome-zarr compress /data/genome-uncompressed.zarr /data/genome-recompressed.zarr --zstd-level 6 --num-workers 8

--chunk-bases must be even, since two bases occupy each byte. Store-level attributes document the encoding and each chromosome array has a logical_length attribute, which removes the one-base padding ambiguity for odd-length sequences.

Each command prints Starting <command>... immediately, then a completion summary with elapsed time, chromosome and base counts, packed-data size, Zarr format, compression mode, and apparent and allocated destination storage. Transcoding commands also print the source store's apparent size.

Python API

from genome_zarr import fasta_to_zstd, decompress_zarr, compress_zarr

fasta_to_zstd("genome.fa", "genome.zarr")
decompress_zarr("genome.zarr", "genome-uncompressed.zarr")
compress_zarr("genome-uncompressed.zarr", "genome-zstd.zarr")