iatro-base-iac

IatroCache (.iac) — a lightweight, high-performance binary container format for offline caching of multimodal medical datasets (image tiles, feature vectors, clinical text, expert tokens, etc.).

This package provides the core container format and readers/writers for .iac files. It is designed for high-concurrency training pipelines, with explicit owned-byte and memory-mapped (mmap) zero-copy read APIs.

Installation

pip install iatro-base-iac

Python API

from iatro.iac import Codec, PackReader, build_pack

Format Layout

[ fixed header        ] 65536 bytes  — magic "IATROC", JSON header (codec, payload_type, offsets, ...)
[ slide table         ] Arrow IPC    — slide_idx / slide_id / patient_id
[ index table         ] Arrow IPC    — caller-defined columns + offset / length / crc32
[ data segment        ] raw bytes    — concatenated payloads, indexed by the index table

Key Technical Features

• Explicit Boundaries & Integrity: Each record carries offset / length / crc32. Payload boundaries are explicit, eliminating the need to scan for framing markers. This works seamlessly for codecs without self-delimiting frames (e.g. raw Brotli).
• High Performance: PackReader.read_payload_views() exposes zero-copy, lock-free mmap views for batch decoding, while read_payload() and read_payloads() always return owned bytes.
• Metadata Flexibility: payload_type and codec are free-form header fields; the low-level container does not interpret the payload bytes directly.

Quick Start (Core API)

Below is an example of writing raw bytes to an .iac pack and reading them back:

import pyarrow as pa
from iatro.iac import build_pack, PackReader

# 1. Define metadata tables
slide_table = pa.table({
    "slide_idx": pa.array([0], pa.uint8()),
    "slide_id": ["s0"], 
    "patient_id": ["p0"]
})

# Offset, length, and crc32 columns are populated automatically
index_table = pa.table({
    "item_id": ["item_a", "item_b"]
})

# 2. Build the cache file
build_pack(
    output_path="out.iac",
    header_json={"payload_type": "raw_bytes", "codec": Codec.NONE},
    slide_table=slide_table,
    index_table=index_table,
    payloads=[b"first_payload_data", b"second_payload_data"]
)

# 3. Read payloads concurrently
reader = PackReader("out.iac")
print(reader.read_payload(1))  # Output: b"second_payload_data"

# Stable owned-data API: always list[bytes]
owned = reader.read_payloads([0, 1])

# Explicit zero-copy API: always list[memoryview], or raises if mmap is unavailable
views = reader.read_payload_views([0, 1])
reader.close()

For large-scale or streaming datasets, refer to build_pack_streaming, build_pack_data_segment, and build_pack_data_segment_from_file.

Pack-level codecs use one public interface. Native byte codecs are selected without a Python implementation wrapper, while user codecs may inherit Codec and register a reconstruction factory:

codec = Codec.create(Codec.ZSTD, level=3)

Reusable concrete schemas are distributed separately in iatro-iac-adapters. The adapter package depends on this core, so one command installs both:

pip install iatro-iac-adapters

Contributing & License

This project is licensed under the MIT License. Codec and Pack contributions are welcome.