IatroCache

IatroCache (.iac) is a lightweight, high-throughput binary cache format for multimodal medical datasets. It combines Arrow metadata with directly addressable payload bytes in one immutable file, supporting image tiles, feature vectors, clinical text, DICOM instances, and project-defined schemas.

The repository contains two public packages:

• iatro-base-iac: the v2 container, native reader/writer, record packs, and codecs.
• iatro-iac-adapters: reusable schemas for tiles, WSI images, teacher features, clinical text, paired text, and patient-level DICOM.

Why IatroCache

• Explicit record boundaries through metadata rather than stream scanning.
• Owned batch reads and an explicit mmap-backed zero-copy API.
• Variable-length records and fixed-width dense matrices in one format.
• Arrow tables for searchable, schema-rich metadata.
• Native checked addressing, I/O, and built-in codecs, with modality policy kept in Python adapters.
• Independent per-record compression for random access and data-loader concurrency.
• Native Raw, Brotli, Zstandard, JPEG XL, JPEG, JPEG 2000/HTJ2K, and JPEG-LS support for clinical text, radiology, and pathology payloads.

Installation

Install the general container API:

pip install iatro-base-iac

Install the reusable medical-data schemas and matching core dependency:

pip install iatro-iac-adapters

Quick start

import pyarrow as pa

from iatro.iac import Codec, PackReader, build_pack

slides = pa.table({
    "slide_idx": pa.array([0], type=pa.uint8()),
    "slide_id": ["slide-001"],
    "patient_id": ["patient-001"],
})
items = pa.table({"item_id": ["a", "b"]})

build_pack(
    "example.iac",
    {"payload_type": "raw_bytes", "codec": Codec.NONE},
    slides,
    items,
    [b"first", b"second"],
)

reader = PackReader("example.iac")
try:
    payload = reader.read_payload(1)
    batch = reader.read_payloads([1, 0, 1])
    metadata = reader.index_table
finally:
    reader.close()

The writer adds offset, length, and crc32 columns for variable records. Batch reads preserve requested order and duplicates and return owned data. read_payload_views() is the separate, explicit zero-copy interface.

Parallel batch decoding

VariableRecordPack combines row gathering with the codec recorded in the package header. Its compatibility API returns one Python object per row:

from iatro.iac import VariableRecordPack

pack = VariableRecordPack("notes.iac")
try:
    records = pack.read_many([100, 4, 100], workers=4)
finally:
    pack.close()

Native Brotli and Zstandard decoding releases the GIL for the complete batch. Raw has no decode work and uses an identity fast path. workers=None uses bounded automatic parallelism, workers=1 is strictly serial, and a larger positive integer limits native work to that count, available logical CPUs, and the number of selected records. Input order and duplicates are preserved. Invalid worker values fail instead of being silently ignored.

Native image codecs keep batch decoding serial until their complete backend paths have a verified parallel-safety contract. Explicit workers>1 therefore fails clearly for JPEG XL, JPEG, JPEG 2000/HTJ2K, and JPEG-LS rather than pretending to provide concurrency.

Contiguous decoded batches

Byte-record consumers can avoid constructing and joining thousands of Python bytes objects:

import numpy as np

pack = VariableRecordPack("notes.iac")
try:
    batch = pack.read_many_contiguous([100, 4, 100], workers=4)
finally:
    pack.close()

consume_buffer(batch.buffer)
first = batch[0]  # read-only memoryview, no record copy
offsets = np.frombuffer(batch.offsets_buffer, dtype=np.uint64)

assert batch.record_count == 3
assert tuple(offsets) == batch.offsets
assert batch.lengths[0] == len(first)

ContiguousRecordBatch.buffer is one owned read-only byte buffer. offsets_buffer is a packed, read-only native-endian uint64 buffer with record_count + 1 entries. The ergonomic .offsets tuple is materialized only when requested, so large native batches do not eagerly allocate one Python integer per boundary. Empty input, zero-length records, repeated rows, and arbitrary row order retain exact boundaries. Batch data and views remain valid after the reader closes.

Performance

The 2026-08-04 native byte benchmark used macOS on ARM64, Python 3.11.15, 1,024 selected records of 64 KiB each (64 MiB decoded), two warmups, and five timing rounds. Zstandard used level 3; Brotli used quality 5 and lgwin=22.

Decoded end-to-end throughput for the contiguous API:

Raw is a zero-decode identity fast path, so worker differences there are measurement noise rather than useful scaling. Representative one-worker stored-byte gathers reached 14.59/19.04 GiB/s for raw, 12.52/22.39 GiB/s for Zstandard, and 12.03/19.89 GiB/s for Brotli, reported as list[bytes]/contiguous encoded gather respectively.

For one 64 MiB call, contiguous output reduced peak RSS relative to the list API: approximately 169 vs 185 MiB for raw, 192 vs 207 MiB for Zstandard, and 254 vs 270 MiB for Brotli at one worker. Results vary by allocator and machine. Accessing the packed buffer for 1,025 offsets took about 0.2 microseconds; materializing the optional Python tuple took about 8 microseconds. The measured compressed paths are therefore decode-bound, not offset-organization-bound.

The byte denominator is explicit: stored-read throughput uses selected encoded bytes; decode and end-to-end throughput use decoded bytes. The benchmark never divides decoded size by pure read time. Reproduce the complete staged timings, Python object construction, contiguous assembly, and fresh-process peak RSS:

conda run -n iac-dev python benchmarks/bench_native_batch_decode.py

Documentation

The IatroCache Wiki covers:

• design and architecture;
• the v2 file format;
• the complete core API and codecs;
• domain adapters for pathology data, clinical text, and DICOM;
• validation, performance and concurrency, and contributing.

Format at a glance

[ fixed 64 KiB header ] magic + version + JSON layout and schema fields
[ slide table         ] Arrow IPC stream
[ index table         ] Arrow IPC stream
[ data segment        ] variable payloads or a fixed-width matrix

The current package version is 0.1.4; the on-disk format version is 2. Those version domains evolve independently.

License

IatroCache is released under the MIT License.