IatroCache

IatroCache (.iac) is a lightweight, high-throughput binary cache format for multimodal medical datasets. It combines Arrow metadata with directly addressable payload bytes in one immutable file, supporting image tiles, feature vectors, clinical text, DICOM instances, and project-defined schemas.

The repository contains two public packages:

• iatro-base-iac: the v2 container, native reader/writer, record packs, and codecs.
• iatro-iac-adapters: reusable schemas for tiles, teacher features, clinical text, paired text, and patient-level DICOM.

Why IatroCache

• Explicit record boundaries through metadata rather than stream scanning.
• Owned batch reads and an explicit mmap-backed zero-copy API.
• Variable-length records and fixed-width dense matrices in one format.
• Arrow tables for searchable, schema-rich metadata.
• Native checked addressing and I/O, with domain decoding kept in Python adapters and codecs.
• Independent per-record compression for random access and data-loader concurrency.

Installation

Install the general container API:

pip install iatro-base-iac

Install the reusable medical-data schemas and matching core dependency:

pip install iatro-iac-adapters

Quick start

import pyarrow as pa

from iatro.iac import Codec, PackReader, build_pack

slides = pa.table({
    "slide_idx": pa.array([0], type=pa.uint8()),
    "slide_id": ["slide-001"],
    "patient_id": ["patient-001"],
})
items = pa.table({"item_id": ["a", "b"]})

build_pack(
    "example.iac",
    {"payload_type": "raw_bytes", "codec": Codec.NONE},
    slides,
    items,
    [b"first", b"second"],
)

reader = PackReader("example.iac")
try:
    payload = reader.read_payload(1)
    batch = reader.read_payloads([1, 0, 1])
    metadata = reader.index_table
finally:
    reader.close()

The writer adds offset, length, and crc32 columns for variable records. Batch reads preserve requested order and duplicates and return owned data. read_payload_views() is the separate, explicit zero-copy interface.

Documentation

The IatroCache Wiki covers:

• design and architecture;
• the v2 file format;
• the complete core API and codecs;
• domain adapters for pathology data, clinical text, and DICOM;
• validation, performance and concurrency, and contributing.

Format at a glance

[ fixed 64 KiB header ] magic + version + JSON layout and schema fields
[ slide table         ] Arrow IPC stream
[ index table         ] Arrow IPC stream
[ data segment        ] variable payloads or a fixed-width matrix

The current package version is 0.1.1; the on-disk format version is 2. Those version domains evolve independently.

License

IatroCache is released under the MIT License.