opencodecs

Native, parallel, cloud-aware codecs for scientific imaging. One unified Codec / Reader / Writer API across compression streams, single images, multi-frame stacks, and chunked containers — with HTTP range-fetch and per-chunk parallelism wired in at the bottom of the stack, not bolted on.

Built for fast modern storage (NVMe, 10 G NAS, S3) where the bottleneck is codec dispatch and per-tile parallelism, not raw I/O bandwidth. Native implementations of every codec — no runtime delegation to imagecodecs — though we use its excellent test suite as a parity reference.

pip install opencodecs

import opencodecs as oc

# 1. Look at any scientific image file
arr = oc.read("scan.czi")              # auto-detect by extension
arr = oc.read("photo.jxl")
arr = oc.read(blob)                    # auto-detect by magic bytes

# 2. Write with the right codec for the data
oc.write("out.jxl", arr, lossless=True)
oc.write("out.zst", b"...payload...", level=10)

# 3. Stream multi-frame / chunked formats
with oc.get_codec("czi").open(path) as r:
    print(r.shape, r.dtype, r.n_frames)
    for tile in r:                     # iter_frames
        ...
    tile5 = r[5]                       # random access

# 4. Fetch tiles of a remote pyramidal TIFF over HTTPS by range request
with oc.open_pyramid("https://example.com/slide.svs") as p:
    region = p.read_region(level=2, y=(1024, 2048), x=(1024, 2048))
    # → 2-3 HTTP Range requests, not a full slide download

# Discovery
oc.list_codecs()                       # capability table
oc.has_codec("avif")

Why opencodecs

Codec capability matrix

All codecs below are native implementations linking against system or vendored C libraries. Build skips cleanly when an optional system library is missing — see INSTALL.md.

Compression (bytes → bytes)

bitshuffle is a filter, not a stand-alone compressor: bit-level transpose that radically improves LZ77 ratios on typed numerical data. Output size equals input size; pair with zstd / lz4. Aliases: bshuf.

deflate aliases: zlib, zlibng. Pass backend="isal" to opt into Intel ISA-L's igzip (~4× faster encode on x86_64; opt-in because output is ~19% bigger). The default backend is auto-selected at build time: libdeflate when present (fastest at default level), else zlib-ng-compat, else the stdlib zlib.

Scientific / numerical-array codecs (ndarray ↔ bytes, self-describing)

These four codecs target typed multidimensional arrays rather than images or raw bytes. The encoded blob carries shape and dtype in its header, so decode(blob) reconstructs the full ndarray without out-of-band metadata.

In fixed-rate mode zfp blocks are individually addressable, so decode_block(data, n) reads one 4x4x4 block without touching the rest (0.0010 ms against 0.209 ms for the whole stream) and a full decode splits the block grid across threads (110 ms → 30 ms on a 67 MB volume). The variable-rate modes have no computable block position and fall back to a whole-stream decode.

| sz3 | ✓ | ✓ | — | abs / rel / psnr / norm | source-built SZ3 | .sz3 | | pcodec | ✓ | ✓ | ✓ | — | source-built pcodec (Rust) | .pco |

Quick guidance:

• pcodec — modern lossless numerical compressor; often beats zstd by 1.5–3× on float / int arrays without a pre-filter.
• b2nd — c-blosc2's multidim layer with shuffle/bitshuffle filters built in; great when you already use blosc2 elsewhere.
• aec — entropy coder used by NetCDF-4 SZIP; lossless integers.
• lerc — fast (lossy or lossless) raster codec used in Cloud-Optimized GeoTIFF, Esri MRF.
• zfp — fast 1D-4D float / int compression with multiple lossy modes (predictable size, accuracy, or precision).
• sz3 — error-bounded prediction-based scientific compressor; often beats zfp at the same error budget on simulation snapshots. Float only (the SZ3 v3 C API doesn't dispatch integer types).

Single-image codecs

htj2k is JPEG-2000 Part 15 (High-Throughput) — same DWT front end as classic JPEG-2000 but ~10-20× faster entropy coding. Used by modern DICOM and remote-sensing pipelines.

jpegls (CharLS) is the lossless / near-lossless predictive JPEG variant standardized as ISO/IEC 14495-1 — the dominant codec in medical-imaging DICOM workflows.

mozjpeg is Mozilla's libjpeg-turbo fork; ~10-15% smaller files than libjpeg-turbo at the same quality. Built only when MozJPEG is on the system (keg-only on Homebrew so it doesn't collide with plain libjpeg-turbo).

rgbe is the canonical Radiance HDR format — float32 RGB shared- exponent encoding for high-dynamic-range photography and physically- based rendering output. ultrahdr is the ISO 21496 gainmap-JPEG format — Android Camera's default since A14 and what iOS 18+ reads natively. Decode dtype controls the output: float16 returns linear BT.2100 HDR; uint8 returns the SDR-tonemapped base JPEG.

Multi-frame / chunked formats

TIFF writer specifics

from opencodecs._tiff_writer import TiffWriter

# Classic TIFF (<4 GiB)
with TiffWriter("out.tif") as w:
    w.write_page(arr, tile=(256, 256), compression="zstd")

# BigTIFF (>4 GiB; magic=43, 64-bit offsets)
with TiffWriter("huge.tif", bigtiff=True) as w:
    w.write_pyramid(levels, compression="zstd", subifds=True)

# COG-style streaming to an unseekable sink (pipe, S3 multipart, HTTP body)
with TiffWriter(sink, streaming=True) as w:
    w.write_stream(pages, total_pages=N, tile=(256, 256), compression="zstd")

Supported encode-side compressions: none, deflate (libdeflate / zlib-ng / zlib auto-detect), zstd, LZW, JPEG, JPEG2000, WebP, JXL, LERC. Horizontal predictor on byte-stream codecs.

OME-TIFF metadata

from opencodecs._ome_xml import write_ome_tiff, Channel

write_ome_tiff(
    "scan.ome.tif", arr_5d, axes="TCZYX",
    physical_size_um=(0.108, 0.108, 0.5),
    channels=[Channel(name="DAPI", emission_wavelength_nm=460),
              Channel(name="GFP",  emission_wavelength_nm=520)],
)

Round-trips through tifffile / Bio-Formats / QuPath. For schema elements outside the 80%-case subset, hand-author OME-XML and pass via TiffWriter's metadata= kwarg.

Remote HDF5

from opencodecs._hdf5_http import open_remote_hdf5, prefetch_hdf5_chunks

with open_remote_hdf5("https://bucket.s3.amazonaws.com/big.h5") as f:
    prefetch_hdf5_chunks(f["img"], np.s_[:1024, :1024])  # 1 syscall, N HTTP
    arr = f["img"][:1024, :1024]                          # all from cache

czi decodes compression types 0 (uncompressed), 5 (zstd), 6 (ZSTDHDR) and 4 (JPEG XR). JPEG XR, which most Zeiss slide scans use, needs the optional _jpegxr extension, built when jxrlib is installed (Homebrew jxrlib, conda-forge jxrlib, Debian/Ubuntu libjxr-dev). The wheels carry it from 0.3.0 on, statically linked on every platform. A JPEG XR sub-block without it raises CziError. Pyramid levels are placed from the slide origin and scale, and overlapping mosaic tiles compose with the higher mosaic index on top, as libCZI and czifile do. The reader exposes metadata_bytes and metadata_xml as lazy zero-copy accessors.

zarr v3 codecs

opencodecs._zarr_codecs registers our compressors as zarr v3 BytesBytesCodecs:

import zarr
from opencodecs._zarr_codecs import OcZstd, OcLz4, OcBlosc2, OcBrotli, OcDeflate

z = zarr.create_array(
    store=..., shape=..., dtype=..., chunks=...,
    compressors=[OcZstd(level=10)],
    zarr_format=3,
)

Performance

0.3.0 against 0.2.0

0.3.0 is mostly a speed release. Each row below ran both versions in fresh processes on the same inputs, alternating which went first, and counts only where both returned identical pixels. Medians in milliseconds, on a 20-core Apple silicon Mac and a 64-core x86-64 Linux workstation.

Rows that name num_workers=8, read_regions or write_many use options new in 0.3.0; they are compared with the only way 0.2.0 could do the same work. Zarr regional reads stay serial unless you pass num_workers (to read_region or OmeZarrArray).

Where it comes from:

• CZI: the directory parses with precompiled structs, pyramid levels are grouped once instead of per level, and each tile is decompressed, unshuffled and written into the output in one native call that releases the GIL once. Separate calls made threads queue for the GIL between steps. Regional reads use a spatial index and decode only the tiles a box touches; read_regions decodes a tile shared by several boxes once.
• Zarr: parallel regional reads are new, with consecutive chunks batched into tasks of about 2 MiB, plain zstd chunks decompressed straight into the output, and v2 blosc chunks decoded through the native blosc2 extension rather than numcodecs, whose binding serializes calls across threads.
• blosc2: each call now has its own compression context. The old path selected the compressor through process-global state behind a global mutex, which serialized every thread and could hand one thread another's compressor (a correctness bug, also fixed).

Across releases

Headline numbers from the latest bench run (bench/run_benchmarks.py --fast, Apple silicon Mac, vs imagecodecs / tifffile / ndstorage):

The PNG encode wins above stack two independent improvements: the libdeflate IDAT accumulator (already shipped) collapses zlib's per-scanline deflate() loop into a single one-shot call, and a per-filter split of libspng's filter_sum hot path lets the compiler autovectorize each branch into NEON/SSE — together they make every PNG-encode workload 1.5–3.1× faster than imagecodecs.

Remote-fetch workloads benefit from read_many (one batched HTTP fan-out + Range coalescing) — on a loopback Range-supporting server, 1024-chunk HDF5 slices land in 7 HTTP requests instead of 1010 (a ~50× request-count reduction; on real-network RTT this translates to 8× wall-clock).

Scientific microscopy CZI (66 MB, 14 sub-blocks of 2000×2000 uint16, ZSTDHDR), single-file warm cache:

See docs/io_patterns.md for the lessons learned about coalesced I/O, mmap vs pread, persistent thread pools, and where parallelism actually pays off. The deflate path is libdeflate when available → zlib-ng-compat → stdlib zlib, auto-detected at build time.

Public API

Top-level dispatch

oc.read(src, *, format=None, **opts) -> ndarray | bytes
oc.write(dest, data, *, format=None, **opts) -> bytes | None
oc.codec_for_path(path) -> Codec | None
oc.codec_for_bytes(head) -> Codec | None

src and dest accept paths, file-like objects, bytes, and memoryview / mmap slices (zero-copy through the codec).

Any of these accept an http(s) URL wherever they accept a path. Formats that can reach storage by offset fetch only the bytes they need by Range request; the whole-codestream formats fetch once, which is the honest thing when every byte is needed anyway.

Which is which is not a list to keep in your head, or in this README where it would rot: capabilities.toml records it per codec and ci/check_capabilities.py verify re-derives every entry from the code on each CI run, so it cannot quietly stop being true.

import tomllib
caps = tomllib.load(open("capabilities.toml", "rb"))["codec"]
[c["name"] for c in caps if c["http"]]      # range-backed over HTTP
[c["name"] for c in caps if c["pyramid"]]   # codecs with a pyramid reader

The runtime manifest covers the codec registry. The companion pipeline catalog also covers direct adapters such as Imaris, OME-Zarr, NDTiff and N5, and maps shared optimization work across all registered codecs. See the implementation plan for priorities, source evidence, correctness prerequisites and acceptance tests. Run python ci/check_pipeline_catalog.py report for the current worklist.

Codec registry

oc.list_codecs() -> list[Codec]
oc.has_codec(name_or_alias) -> bool
oc.get_codec(name_or_alias) -> Codec

Codec interface

Each codec exposes:

codec.name            # "czi"
codec.file_extensions # (".czi",)
codec.has_native      # True for everything we ship
codec.can_encode / codec.can_decode
codec.multi_frame / codec.chunked / codec.streaming_decode / codec.parallel_decode
codec.supported_dtypes / codec.supports_color

codec.signature(head_bytes) -> bool
codec.encode(data, *, dest=None, **opts) -> bytes | None
codec.decode(src, **opts) -> ndarray | bytes
codec.open(src, **opts) -> Reader        # multi-frame / chunked

Reader interface (multi-frame / chunked)

reader.shape       # (n_frames, *frame_shape)
reader.dtype
reader.n_frames
reader.is_chunked  # True if [idx] random access works
reader.iter_frames()
reader.read()      # full eager decode
reader[idx]        # random access (chunked formats only)

CZI reader additionally exposes:

reader.entries                  # list[CziSubBlockEntry] — sub-block metadata
reader.metadata_bytes           # raw UTF-8 bytes (lazy + cached)
reader.metadata_xml             # decoded str (lazy + cached)
reader.subblock_metadata_bytes(i)

HDF5 reader additionally exposes:

reader.dataset_names            # all numeric datasets in the file
reader.select(name)             # switch to a different dataset

Streaming-reader examples

1. Fetch a region of a remote Aperio whole-slide TIFF

import opencodecs as oc

# Pyramidal SVS (Aperio) hosted on S3 / any HTTPS endpoint with Range support.
with oc.open_pyramid("https://example.com/slide.svs") as p:
    print(p.levels)               # [(80000, 60000, 3), (40000, 30000, 3), ...]
    region = p.read_region(level=2, y=(1024, 3072), x=(2048, 4096))
    # Total HTTP traffic: ~6 Range requests covering only the tiles
    # that intersect this 2048×2048 bbox — typically 200 KB–2 MB,
    # not the 4 GB whole slide.

The pyramid reader auto-detects the best level for the requested region, fetches only the intersecting TIFF tiles via HTTP Range, and assembles the output in-memory. Works the same on local files, NFS, SMB, S3, or any range-capable HTTP server.

open_pyramid dispatches on extension for TIFF/COG/SVS, OME-Zarr, CZI, Imaris, JPEG, JPEG 2000 and HTJ2K, and two whole-slide formats whose pyramid is not one file:

# Olympus / Evident CellSens. The .vsi is an index; the tiles live in
# a sibling _NAME_/stackN/frame_t*.ets. Only the tiles the box covers
# are decoded.
with oc.open_pyramid("slide.vsi") as p:
    p.shapes            # ((9367, 8022, 3), (4684, 4011, 3), ... 6 levels)
    tile = p.read_region(0, y=(1000, 1256), x=(2000, 2256))

# DICOM VL Whole Slide Microscopy. A slide is a SERIES of instances,
# one per resolution, so this takes a directory rather than a file --
# no extension can imply that, hence the explicit format=.
with oc.open_pyramid("study/slide_dir", format="dicom") as p:
    overview = p.read_region(p.best_level_for(max_pixels_y=1024))

2. Convert a multi-level pyramid to OME-Zarr v3 sharded

import opencodecs as oc

with oc.open_pyramid("input.ome.tiff") as p:
    levels = [p.read_region(level=i) for i in range(len(p.levels))]

oc.write_omezarr_pyramid(
    "output.zarr",
    levels,
    chunks=(512, 512),
    shards=(2048, 2048),         # 16 chunks per shard, one file each
    compressor="zstd",
    zarr_format=3,
)
# 1 file per shard on disk instead of 1 file per chunk; per-chunk
# random access still works via Range fetches into the shard.

For data going to S3, sharded Zarr v3 cuts your PUT and LIST costs by 1–2 orders of magnitude vs unsharded chunks while preserving per-chunk random-access via HTTP Range — the reader above understands the shard index automatically.

3. Fast JPEG XL thumbnails (native progressive decode)

import opencodecs.jxl as jxl

# downsample=8 uses libjxl's native progressive decoder — stops at
# the DC pass without reconstructing full-resolution pixels.
thumb = jxl.read("scan.jxl", downsample=8, subsample="center")
# 4Kx4K input → 512x512 ndarray in ~28 ms on macOS arm64
# (vs ~40 ms for a full decode), positionally centroid-correct
# so SVG / GL renderers don't get a ½-block shift.

# For a partial JXL bitstream usable as a tiny browser-direct
# thumbnail (works in Safari + modern Chrome):
prefix = jxl.thumbnail_bytes("scan.jxl")
# → ~85 KB out of a 3.5 MB source for a 4Kx4K image

Install

pip install opencodecs

Wheels are published for CPython 3.10 to 3.13 on macOS 15+ (arm64), Linux (x86_64 + aarch64, manylinux_2_28), and Windows (amd64). Every platform ships the same 39 compiled extensions, bundling libjxl, libavif, libheif, libwebp, libdeflate, c-blosc2, CharLS and friends, so no system dependencies are needed. Intel Macs and macOS releases older than 15 install from the sdist instead.

For a source install, system development headers, or to build a tuned local libjxl, see INSTALL.md. Wheel publishing runs through docs/publishing.md.

# Source install — auto-detects system libs, source-builds libjxl
git clone https://github.com/kevinjohncutler/opencodecs.git
cd opencodecs
pip install -e .

The build skips cleanly for any system library that's missing — useful extensions still build, missing ones print a one-line notice. libjxl 0.11.2 is auto-built from source via bench/build_libjxl.sh and cached under ~/Library/Caches/opencodecs/ (macOS) / ~/.cache/opencodecs/ (Linux). See INSTALL.md for the rationale (Homebrew/apt builds are 0.5-0.7× slower than a tuned -O3 + LTO build).

Status

• v0.3.0 on PyPI (September 2026). Every wheel carries the same 40 compiled extensions, and ci/check_wheel_contents.py fails the release build if one goes missing.
• About 4,000 tests locally, including a 40-dataset conformance corpus; CI runs the corpus-independent suite on macOS, Linux and Windows for Python 3.10 and 3.13.
• Native readers and writers for the common scientific containers (TIFF, BigTIFF, OME-TIFF, CZI, NDTiff, HDF5, JXL, FITS, OME-Zarr v2 + v3 sharded), and pyramid readers for TIFF/COG/SVS, OME-Zarr, CZI, Imaris, DICOM VL Whole Slide Microscopy and Olympus VSI/ETS
• Shared source adapters support paths, buffers, file objects and remote URLs. Indexed readers use checked range requests, covering caches and adaptive read-ahead; whole-stream codecs retain their documented eager input paths.
• Multi-frame AVIF, animated WebP and GIF decode to a frame stack; HEIF exposes every top-level image
• Compression backend auto-detect (libdeflate → zlib-ng-compat → stdlib)
• tifffile_patch opt-in shim reroutes tifffile's codec dispatch through opencodecs for users who want only a partial swap
• capabilities.toml records what each codec actually supports, checked against the built extensions, with no open gaps

Deferred work (see docs/TODO_DEFERRED.md):

• CCITT Fax3/Fax4 encode: legacy fax, zero scientific users
• JPEG XR encode (decode exists for CZI; see above)

GIF and JPEG XL destination writers now drain encoded frame output before close. The shared pipeline also supplies bounded independent-piece scheduling, seven stateful byte-codec iterators, native PNG row sessions, selective native sources, and checked caller-owned destinations.

Shared streaming, buffering, and overlap mechanisms and remaining integration work are documented in the pipeline catalog.

See CHANGES.rst for release history.

License

BSD-3-Clause; see LICENSE.

Vendored source, the Cython declaration files derived from imagecodecs (BSD-3-Clause, Copyright (c) 2008-2026 Christoph Gohlke), and the codec libraries bundled into the binary wheels each retain their own license. The full inventory is in THIRD-PARTY.md.