Loaderx

A compact, high-performance persistent record store with zero-copy batch gathering and transparent per-record compression, designed for single-machine AI training and serving pipelines

pip install loaderx

Wheels are published for Linux (glibc ≥ 2.17 and musl, x86-64 and arm64), macOS (≥ 11.0, Intel and Apple Silicon) and Windows (x64 and arm64). The bindings use cffi in ABI mode, so nothing links against the CPython ABI and one wheel per platform serves every supported Python.

Design Philosophy

loaderx is built around several core principles:

1. A pragmatic approach that prioritizes minimal memory overhead and minimal dependencies.
2. A strong focus on single-machine training workflows.
3. We implement based on NumPy semantics, persisted through the Zrecord storage runtime.
4. An immortal (endless) step-based data loader, rather than the traditional epoch-based design—better aligned with modern ML training practices.
5. Dense and ragged are separate contracts, and the loader serves both. A dense stream stacks into one array per batch; a ragged one comes back as a list. Neither is padded, and equal length is never treated as a special case of variable length.

Quick Start

from loaderx.utils import from_numpy
from loaderx.dataset import DenseDataset
from loaderx.dataloader import DataLoader

from_numpy('train_data', np.load('data.npy', mmap_mode='r'))
from_numpy('train_label', np.load('label.npy', mmap_mode='r'))

loader = DataLoader({'data': DenseDataset('train_data'), 'label': DenseDataset('train_label')},
                    transform=lambda batch: batch)

for i, batch in enumerate(loader):
    if i >= 256:
        break

print(batch['data'].shape)
print(batch['label'].shape)

A batch is a dict {name: values}: each value is the stacked (batch_size, *item_shape) array for that stream. Every stream is gathered at the same indices, so record i lines up across them. The transform callback is the collate step — reshape, cast, stack — where values is the plain dense batch ready for the model.

Converting a NumPy tensor

import numpy as np
from loaderx.utils import from_numpy
from loaderx.dataset import DenseDataset

from_numpy('train_data', np.load('data.npy', mmap_mode='r'))
from_numpy('train_label', np.load('label.npy', mmap_mode='r'))

# many small, similar records compress far better against a trained dictionary:
from_numpy('train_data', images, codec='zstd_dict')

One record per slice along axis 0; a 1-D array (the usual shape of a label set) becomes a dataset of scalars. The conversion streams in bounded chunks, so an mmapped array is never fully materialized. Open the result with :class:DenseDataset:

ds = DenseDataset('train_data')

The store carries a single schema.msgpack recording the kind ("dense" or "ragged"), the per-sample dtype, and the shape contract — one item_shape for a dense store, one shape per record for a ragged store. That is the only loaderx-level metadata. Everything beneath it is plain Zrecord, reachable through loaderx.zrecord.Zrecord when you want raw byte records (ragged samples, pre-encoded images) instead of tensors.

Records

One store engine, two kinds of view. from_numpy writes a dense store where every record is exactly one row of the recorded item_shape; reads are fixed-stride gathers and the batch shape follows from the schema, so no per-record metadata is touched:

import numpy as np
from loaderx.utils import from_numpy
from loaderx.dataset import DenseDataset

from_numpy('data', np.arange(64, dtype=np.float32).reshape(8, 2, 4))
ds = DenseDataset('data')
ds[0, 5, 2]                          # (3, 2, 4) — a record set, shape straight from the schema

A dataset is a collection of records, not an ndarray, so ds[0, 5, 2] is a record set — never ds[0][5][2]. A scalar selects one record; negatives wrap from the end. The ragged example below reads the same way.

from_iterator writes a ragged store of variable-length records. It is a separate contract: :class:RaggedDataset hands back a list of arrays, so a loader never has to carry row_splits around. dtype is unified and explicit; each record keeps its own shape, recorded per record as it is written and restored exactly on read — so records may differ in shape arbitrarily, and nothing is ever inferred from the source (an iterator can't tell you what its later records look like). Densifying a list into a dense batch is the model's call — a plain numpy loop, wherever you need it:

from loaderx.utils import from_iterator
from loaderx.dataset import RaggedDataset

seqs = [np.arange(L, dtype=np.int32) for L in (3, 1, 4, 1, 5)]
from_iterator('tokens', seqs, np.int32)   # dtype explicit; each record keeps its shape
rs = RaggedDataset('tokens')

records = rs[0, 2, 4]                # list of ndarray — one per record, exact shapes
lengths = np.array([len(r) for r in records])
padded = np.zeros((len(records), lengths.max()), dtype=records[0].dtype)
for i, r in enumerate(records):
    padded[i, :len(r)] = r             # (B, max_len) — your policy, your loop

A DataLoader works with dense streams, so collation is just the transform — a batch dict in, a batch dict out:

def collate(batch):
    return {'input_ids': batch['tokens'], 'label': batch['label']}

loader = DataLoader({'tokens': dense_tokens, 'label': labelset}, batch_size=32,
                    transform=collate)

Writable datasets

The dataset classes are the general user-space scheme over zrecord: they are read-write, and append is explicit — one container of records, one batch, one native call. Nothing is buffered, inferred, or compressed for you.

from loaderx.dataset import DenseDataset, RaggedDataset

ds = DenseDataset('mnist/x', dtype=np.uint8, item_shape=(28, 28))  # schema written now
ds.append(images[i:i + 1024])        # (B, 28, 28) → one batch, returns first index
ds.append(single_image[None])        # one sample is batch_size 1 — add the axis yourself
ds[:4]                               # read path is unchanged

tok = RaggedDataset('tokens', dtype=np.int32)     # each record keeps its own shape
tok.append([seq_a, seq_b, seq_c])    # list in, list out — `tok[0, 2]` returns a list

The schema is declared at construction and written to schema.msgpack immediately; append validates every record against it and errors loudly. The dictionary is the one thing kept out of append: train_dict is explicit and separate, and a zstd_dict store refuses to append until it is called.

The store's maintenance pass-throughs are on the same object, normalized like reads (scalar, slice, array, bool mask):

ds.delete(np.arange(0, len(ds), 2))   # swap-last: index space stays dense, survivors reindex
ds.stats()                            # {'records', 'live_bytes', 'chunk_bytes', 'reclaimable'}
ds.compact()                          # reclaim the deleted bytes, in place — offline only

Deletion and compaction move records, so a multi-stream index you keep yourself must tolerate it — the guarantee is only that the live records are exactly 0..len(ds).

Codec notes

"zstd" compresses each record independently with plain zstd (level 3). Use it for general-purpose compression — it is fast and the default.

"zstd_dict" trains a shared dictionary on a sample of the data before writing any record, then compresses every record against it at level 19. The dictionary captures structure shared across records that per-record compression cannot see — a large win for many small, similar records (image tiles, token sequences).

The dictionary's cost is a cache footprint: every record's decompression references the shared dictionary window, so a larger dictionary means more cache misses on gather — the path a loader pays forever. Three named tiers (loaderx.dataset.DICT_TIERS) preset the whole tradeoff — the dictionary size and how much data trains it — so a caller picks a tier, never a number:

The sample is the byte budget the dictionary trains on (a strided subset of the records), so each tier costs the same training time whatever the record size. At realistic image scale (768 KiB records) the tiers converge — on the earlier measurement box's structured data "balanced" and "max" both gather ~1.6 GiB/s at a 1.74x ratio — because a dictionary is a small fraction of a large frame. The tiers still matter at small record sizes, where the dict is most of a record and "max" trades gather throughput for ratio.

"zstd_dict" records can only be read from a store that has the dictionary (dict.zr). The dictionary is loaded on open and shared, lock-free, across all reader threads.

A dictionary must train on the settled, complete data. :func:train_dict is the standalone, manual training step — a numpy array, an iterable of records, or a dataset all train the same way, sized by a tier — and its bytes are handed to a store-writing path via dict_bytes. zstd_dict never trains by itself: a write without a dictionary is an error. A stream cannot train its own dictionary, but it can write with one trained on the settled data:

from loaderx.utils import train_dict, from_iterator, from_numpy

# train once on the settled data — a standalone, reusable artifact
d = train_dict(settled_array, tier="balanced")

# then any store can write zstd_dict with it, including a stream
from_iterator('tokens', token_generator, np.int32, codec='zstd_dict', dict_bytes=d)
from_numpy('data', data, codec='zstd_dict', dict_bytes=d)

Changing an existing store's codec is a rewrite, not a store mutation: create a new store with the new codec and copy the records in index order. Index order is what keeps multi-stream alignment, and the copy loop is a few lines with the dataset layer — which is why no dedicated recode path exists:

from loaderx.utils import train_dict
from loaderx.dataset import DenseDataset

CHUNK = 1 << 16
with DenseDataset("src") as s, \
     DenseDataset("dst", dtype=s.dtype, item_shape=s.item_shape,
                  codec="zstd_dict") as d:
    if not d.has_dict():                       # zstd_dict needs a dictionary
        d._store.install_dict(train_dict(s))   # train one on the source store
    for start in range(0, len(s), CHUNK):
        d.append(s[start:start + CHUNK])       # records copied in index order

RaggedDataset is the same shape — swap the constructors and dtype is all the destination needs. dst must not already hold a store.

Important: Do not use "zstd_dict" while data is still changing (through deletes or compaction below the Python layer). The dictionary captures a snapshot of the data; training it before the data settles wastes compression. Train the dictionary once preprocessing is complete and the content is final.

Multi-stream datasets (MSDataset)

A zrecord store is one stream; a training sample is usually several streams (skeleton + label + id, tokens + label, ...). A multi-stream dataset is just a directory whose immediate children are stores. MSDataset wraps them and guarantees the one thing that matters: index alignment — record i lines up across every stream, append writes every stream at the same indices, delete removes the same indices from every stream, so alignment survives. No new storage format, no manifest, no bundle-level batch API.

from loaderx.utils import from_numpy, from_iterator
from loaderx.dataset import MSDataset
from loaderx.dataloader import DataLoader

root = "xsub/train"
from_numpy(root + "/joint", joint)     # the streams are ordinary zrecord stores
from_numpy(root + "/label", label)
from_iterator(root + "/token", iter(seqs), np.int32)

ds = MSDataset(root)                   # wrap + verify they hold the same count
ds["joint"][0, 5, 2]                 # the stream's own index forms apply
ds.append({"joint": b, "label": l})    # same batch size, all streams, once
ds.delete([0, 5])                      # same indices, all streams, still aligned

loader = DataLoader(ds.streams, batch_size=256)   # hand the streams to a loader
batch = next(loader)                      # {name: values}, index-aligned

Equal-length and variable-length are both just stores — DenseDataset (one fixed-shape record per sample) and RaggedDataset (variable row count per record). The loader does not interpret either: it fetches by index and packs a dict, so a dense stream's batch value is the stacked (B, *item_shape) array and a ragged stream's is a list of per-record arrays. No padding is imposed — densify to a fixed shape however the model needs (a plain numpy loop), or reshape/stack in the transform collate.

CPU → GPU transfer

loaderx hands over CPU batches; getting them to the accelerator is the transform's job — the one place your framework is already imported. The batch dict is a plain {name: numpy array}, zero-copy on the way out, so a device transfer is one call per stream:

import torch

device = "cuda:0"
def to_device(batch):
    return {k: torch.from_numpy(v).to(device, non_blocking=True)
            for k, v in batch.items()}

loader = DataLoader(streams, transform=to_device)
for batch in loader:
    model(batch)                        # already on device

A non_blocking=True copy is genuinely asynchronous only when its source is pinned. loaderx does not pin memory for you — pinning is framework-owned (torch's .pin_memory(), CUDA's cudaHostAlloc), and a vendor-free core stops exactly at the CPU batch. Pin in the transform what you copy:

def to_device(batch):
    return {k: torch.from_numpy(v).pin_memory().to(device, non_blocking=True)
            for k, v in batch.items()}

JAX is the same shape — jax.device_put is already an asynchronous handoff on GPU:

import jax

def to_device(batch):
    return {k: jax.device_put(v) for k, v in batch.items()}

The transfer runs on the transform stage and never touches loaderx internals: the copy overlaps the next batch's gather/transform, and any pinned pool is the caller's to own and reuse. This is the entire H2D answer — there is no pin= hook or device backend, because the only unified thing a multi-framework loader can own is the CPU batch.

For practical integration examples, please refer to the Data2Latent repository

Benchmarks

scripts/bench.py measures zrecord and loaderx against the alternatives in the index sampler, the record store, and the full data loader — each through the binding a client actually uses, so cffi, the GIL and the NumPy allocation are all inside the timings. One run is enough: this is a qualitative horizontal comparison, sized to a realistic image workload (625 MiB of 256 KiB records) that finishes python3 scripts/bench.py all — which prints this machine block first — in under half a minute. Warm page cache.

Machine — one box, an LXC container on a server (AMD EPYC 7303, 16 cores / 32 threads):

The container sees all 32 threads under a 32 GiB cgroup limit. The benchmark runs on the ordinary page cache.

1. Sampler — index generation on its own, IID (with replacement), 1M index space, against NumPy's modern API. The µs-scale figures fluctuate with box load; the stable signal is the ~1.9x margin, roughly flat across batch sizes.

2. Store — Zrecord against the alternatives: random batch gather, 2500 records, batch 256, structured (mildly compressible) data. A single 256 KiB sample (512×512 single-channel, a realistic image record) is the horizontal comparison; other record sizes are a --shape away. hdf5-gzip, arrayrecord, torch and grain are installed and run in the default tables.

Realistic records — 256 KiB per record, 64 MiB per batch:

write is page-cache ingestion — the store is built and left open, exactly like np.save and the other backends, which defer durability to the kernel. A zrecord is append-only, so this is one pass over the frontier; at 256 KiB records the 16-byte table entry per record is negligible and raw write leads .npy. Durability is the explicit sync()/close — earlier write numbers included zrecord's close-time fsync while the competition did not, which compared durable writes against lazy ones. The absolute write figures fluctuate with box load on this shared host; the raw store's margin over .npy is the stable part.

gather is fully page-cache-warmed — the benchmark sweeps every record once before timing, so it measures the pure access path (memory), not first-touch page faults or disk. At 256 KiB records the batch (64 MiB) exceeds L3, so these numbers are DRAM-bandwidth-bound for every backend; the 12 KiB table that predates the record-size bump was an L3-fit artifact. Fully warm, zrecord-raw leads npy-mmap by ~2.8x on this box — its gather fans out across every core while numpy's fancy index is single-threaded — and the gap widens where single-threaded copy is slower.

3. Loader — the full input pipeline end to end (sample, fetch, collate, hand over a batch), same workload, 4 workers, batch 256 (64 MiB). The peak RSS column (added with the uniform-codec rewrite) is the highest resident set size of the whole process tree while batches are flowing. torch, grain and arrayrecord are installed and wired into the defaults, and all four loaders run at the full batch.

At 64 MiB per batch the loader is DRAM-bound, not sampler-bound — the per-batch gather (zstd ~3.9 GiB/s, raw ~5.4 GiB/s) is the whole story, and the prefetch threads keep it at store-gather speed while the Python side collates. The memory is the prefetch buffers plus the two dataset handles (raw + label stores over the same backing data). loaderx prefetches in threads inside one process, so workers share one interpreter, one numpy and one set of gather buffers; torch and grain run a worker process per prefetch thread, which is most of their RSS (torch's peak also includes the shared-memory collated batches). grain's batches/s is the pipeline's floor here, ~5x below loaderx-raw.

Free-threaded Python. loaderx targets free-threaded builds (no GIL), and the key sections are also measured on 3.14t — the question is whether anything gains when nothing is GIL-limited. These are opt-in runs (a second interpreter and --transform augment); they are not part of the default all:

• Sampler — zsampler draws ~1.9–2x faster than free-threaded numpy, the same margin as on the GIL build: a batch is one Zig call either way.

• Store — the free-threaded numbers track the GIL table within ~10%. zrecord-raw gathers 5.1 vs 5.4 GiB/s (still ~2.8x npy-mmap), zrecord-zstd 3.9 vs 3.9 GiB/s. Notably, the alternatives do not gain from the build: blosc2's free-threaded wheel re-enables the GIL to load its extension, so it runs exactly as on the GIL build. Random gather, 256 KiB records:

  store	CPython 3.14 (GIL)	free-threaded 3.14t
  zrecord-raw	5383 MiB/s	5123 MiB/s
  zrecord-zstd	3943 MiB/s	3894 MiB/s
  npy-mmap	1737 MiB/s	1812 MiB/s
  hdf5	1016 MiB/s	1039 MiB/s
  blosc2	336 MiB/s	342 MiB/s
  tensorstore	116 MiB/s	139 MiB/s

• Loader, identity — unchanged, ~60 batches/s (loaderx) on both interpreters at 64 MiB per batch.

• Loader, CPU-heavy transform — the one place the free-threaded build matters. A transform runs on the prefetch threads, so the GIL serializes it on standard CPython — the case where worker processes win, and the reason for the free-threaded build. --transform augment (a Python-bound per-sample loop), 4 workers, loaderx vs torch (which has no free-threaded wheel, so its row is the GIL build's worker processes):

  loader	CPython 3.14 (GIL)	free-threaded 3.14t
  loaderx	24.3	41.1 batches/s
  loaderx-raw	22.3	43.3 batches/s

  Free-threaded still wins — the transform parallelizes across the prefetch threads — ~1.7x on loaderx and ~1.9x on raw at 256 KiB. The margin is narrower than at 12 KiB records (where a batch fit in cache and the transform was the whole cost) because the 64 MiB gather is DRAM-bound, but wider than at the 768 KiB scale, where the pipeline was even more DRAM-bound.

Conclusion — why the numbers look like this.

Every hot path is one native call. Zsampler draws a whole batch of indices, zrecord gathers a whole batch and decompresses it, in a single cffi call into Zig with the GIL released and the batch copied straight into its destination buffer. The contenders do the same work one record at a time from Python. That one fact runs through all three tables: the sampler's margin is roughly flat across batch sizes (a batch costs one call either way, so the per-index work is what divides them), and the store gather column is where one-record-per-call costs the most. The speedup is not bought with distribution shortcuts: the IID draw is unbiased like NumPy's (Lemire with rejection, so uniformity costs nothing over a real index space).

The layout matches what a training loader does. zrecord is built for random record access — the record table is indexed in O(1), a dense store gathers at a fixed stride. Array stores are built for contiguous scans, so a scattered batch — exactly what a loader reads — fights their layout. A raw gather is a fan-out across every core, where NumPy's fancy index is one thread; on the multi-core memory subsystem that closes the gap to an uncompressed .npy which does no per-record work at all.

Compression is in the kernel, and there is one codec. zrecord-zstd is not "storage plus a codec": the layout, the multi-core decompress and the GIL-free copy are one path, so turning compression on costs part of a margin, not an order of magnitude — ~3.9 GiB/s here, ~11x ahead of the other compressed stores (116–336 MiB/s) at the same ~1.1x ratio. The modest ratio is the data, not the format: these samples are barely compressible, and on a smooth image set plain zstd reaches 7.6x and the dictionary (at the "max" tier) 16.5x.

The loader gap is architecture, not storage. loaderx uses threads and never ends an epoch, so a step pays no IPC and never waits on an epoch boundary; torch restarts per epoch with worker processes. That is most of the ~1.6x over torch and ~3.8x over grain here. Storage also differs per loader — each reads from what it was built for — so the loader table is a different comparison than the store table, not a rerun of it. The one crack in the thread model is a CPU-heavy transform, which the GIL serializes — that is exactly what free-threaded Python removes, so loaderx is developed and benchmarked against free-threaded builds first.

What these numbers do not claim. Everything runs with a warm page cache: this measures the access path, not cold storage or disk. on disk is allocated blocks, so zrecord's sparse preallocation is not charged to it while TensorStore's sharding is; both are layout, not compression. write is page-cache ingestion with durability deferred, matching the other backends — zrecord's own durability (sync/close) is a separate, explicit cost that neither this table nor the competition pays. This is a single qualitative pass — the µs-scale sampler timings and the loader batches/s fluctuate with box load (on these 16 cores the compressed loader trails the raw one by ~25–30%), so treat the absolute numbers as ballpark and the cross-backend margins as the signal.

Real-data verification: NTU RGB-D skeletons

The tables above are synthetic workloads. As a ground-truth check, loaderx was run end to end on NTU RGB-D skeleton data — 114,480 raw .skeleton files, 120 action classes, 25 joints — processed into the ST-GCN N C T V M layout (per-sample (3, 300, 25, 2) float32) for the xsub/xview protocols. The .npy outputs of the standard preprocessing pipeline were treated as ground truth; every number below comes from a machine identical to the Benchmarks box.

Correctness — the read path is bit-exact against the ground truth:

• Full scan of all 228,356 records (joint float32 + label int64, all four splits) through DenseDataset: byte-for-byte identical to the reference npy.
• A DataLoader over joint + label + an index stream, run under all three sampler modes (sequential, iid, cyclic): every received batch is bit-exact to the ground truth at its own declared indices, and the streams stay index-aligned.
• Sampler semantics hold on real index spaces: sequential walks in order, cyclic draws a full cycle without replacement, iid is deterministic per seed.

Storage — zstd on this data:

Sizes above are for one split (xview/val, 38,132 records); across all four splits the zstd joint stores total 4.97 GB against 41 GB of raw npy (~8x).

Throughput (180 KB per record, warm page cache, 12 physical cores):

zstd decompression reads ~8x fewer bytes than raw storage, so the compressed store gathers faster than the raw one (zstd 4587 MiB/s vs raw 1792 MiB/s on the same split).

The npy intermediate is optional. from_numpy is chunked append under the hood, so the whole npy staging step can be skipped: parse the skeleton files in parallel and feed DenseDataset.append the fixed-shape chunks directly. This writes the store in one pass (no npy, no second read), and the resulting store is byte-for-byte the same size as the from_numpy equivalent — verified record by record against the npy ground truth.

Current Limitations

• Single-host only; multi-host training is not supported.
• A single sample must be at most 2 GiB (2^31 bytes). There is no fixed record count: length is a u64 and the record table grows on demand, so how many records a store holds is bounded by its total chunk capacity (up to 2^64 bytes) divided by the average record size — e.g. roughly 2^44 records at 1 MiB each, 2^33 at 2 GiB each.
• Metadata is read and written as the host's struct layout, so a store carries the host's byte order and is not portable to a machine of the opposite endianness. Every published platform is little-endian, so this only matters if you build for one yourself.

Build

zig build                       # host shared objects, into loaderx/lib/
zig build test                  # Zrecord suite, in both Debug and ReleaseFast
python3 tests/test_loaderx.py   # Python layer, against whichever build is importable
python3 scripts/bench.py        # throughput, against other stores

The Zig side is tested for behaviour only; throughput is measured from Python, through the binding a client actually uses. scripts/bench.py runs in three layers (sampler, store, loader, or all) and skips contenders that are not installed. See Benchmarks.

Publishing

Zig cross-compiles every target from one machine, so releases need no CI matrix:

zig build dist                    # every platform, into zig-out/dist/<wheel tag>/
python3 scripts/build_wheels.py   # one wheel per platform, plus the sdist

The dist directories are named after their Python wheel platform tag, so the tag mapping lives in exactly one place (dist_targets in build.zig). glibc and macOS minimums are pinned in the target triple, which is what makes manylinux_2_17 and macosx_11_0 honest rather than aspirational. Each wheel is checked after packing: it must carry this platform's libraries and no others.

The sdist ships sources only. Installing from it runs zig build through setup.py, so it needs the Zig compiler; wheel users never hit that path.

Zsampler

Index Generator: a high-performance sampler implemented in Zig. Every mode is a pure function of (seed, step), so a run resumes exactly by seeking to a step — there is no epoch to track, in keeping with the endless step-based loader.

1. Sequential — traverse the index space in order through a fixed-size sliding window, treating the space as a circular queue so the tail never truncates.
2. IID — draw each index uniformly at random with replacement. Unbiased (Lemire with rejection), matching NumPy. Simplest, but coverage is uneven over any short run.
3. Cyclic — without replacement, round-robin. Each cycle traverses a fresh permutation of the whole index space, so within a cycle every record appears exactly once and no batch repeats an index — coverage is even by construction, which keeps how often each sample is seen uniform. The permutation is a stateless bijection (a small Feistel network over the index space, brought into range by cycle-walking), so a million-record shuffle materializes nothing the size of the dataset and reshuffling each cycle is free. Every batch is exactly batch_size: an endless step-based loader has no final partial batch to special-case, so when batch_size does not divide the length the cycle's remainder is dropped — a different remainder each cycle, since the permutation changes, so every record is still reached over time.

Zrecord

A record-based data runtime, focused on delivering extreme throughput and low latency.

1. Zrecord is an unordered physical store made of N records. Records are independent and carry no ordering, so every index and slice operation is equivalent to a gather.
2. Zrecord hands the client a dense index space: the live records are always exactly 0..N. Deletion preserves that by swapping the tail into the hole, which means an index is stable only until something is deleted. A multi-stream dataset is purely a client-side notion; the client keeps its own stream index table pointing at records.
3. Zrecord returns byte arrays. Type interpretation is the client's job.
4. The IO model (append | read | delete) is batch-oriented and shape-agnostic. A record is a packed byte range located by an offset; there is no notion of fixed vs variable length, and a single-record operation is just the batch_size == 1 case. Equal length and single records are special cases, not parallel code paths.
5. Zrecord owns its memory internally — allocation and release are explicit.
6. A store has exactly one codec, fixed at creation and immutable afterwards — matching the dataset semantics above it, where the schema records a single codec. Every record is compressed and decompressed independently:

| tag   |  name     | algorithm                              |
|-------|-----------|----------------------------------------|
|   0   |  raw      | none                                   |
|   1   |  zstd     | zstd (plain, level 3)                  |
|   2   |  zstdict  | zstd with a trained dictionary (level 19) |

7. Compression is transparent to the client:
   • Compression runs concurrently across all cores. A compressed store never falls back to raw: each record is stored as the codec's output, even when an incompressible record's frame is larger than its input — write raw if the data does not compress.
   • Decompression writes straight into the caller's destination memory (gather), with no intermediate buffer and no extra copy.
   • zstd is the one transparent codec — faster than Deflate at both ends and a better ratio, so there is no reason to carry a second. It is vendored C, built for every platform by Zig, so the one-wheel-per-platform story is unchanged.
   • zstd_dict additionally trains one dictionary on a sample of the data (stored as dict.zr) and compresses every record against it. Because each record is still independent, random access is unchanged — but the dictionary carries the structure shared across records, which per-record compression cannot see. On many small, similar records (image tiles, token sequences) this is a large win: a smooth-image set that plain zstd takes to 7.6x compresses 16.5x with a large dictionary. The dictionary is loaded once on open and shared, lock-free, across all reader threads. The dictionary size is chosen from the DICT_TIERS presets (see Codec notes).
   • A zstd_dict store needs its dictionary to read every record; a raw or zstd store never touches a dictionary even if one is present.

Persistence format

Zrecord storage is metadata plus chunked data. The extension is the type: .zr files are store-global singletons, .loc files are record-table segments, .chunk files are record data:

zrecord/
  ├── meta.zr      header (global state)
  ├── dict.zr      zstd dictionary (only in dict stores)
  ├── 0.loc        record table segment [0, 2^28)
  ├── 1.loc        record table segment [2^28, 2^29)
  ├── 0.chunk      record data
  └── 1.chunk

Metadata (meta.zr + {id}.loc)

Files are read and written positionally — pread/pwrite at computed offsets, no mmap. The header is a bit-packed struct and a .loc segment an array of 16-byte RecordLocs, exactly as wide as they declare, so a location is one pread/pwrite of 16 bytes at a computed offset and there is no serializer anywhere in the code. Every header field is byte-aligned (no bit fields cross a byte), so the packed header reads as plain memory.

The record table is partitioned into {id}.loc segments so it can grow by appending a segment instead of reserving the maximum. The id→segment mapping is pure arithmetic — seg = idx >> 28, off = (idx & (2^28−1)) × 16 — so a segment needs no per-record bookkeeping. A segment is immutable once published, so each mapping's base never moves and lock-free readers are safe to index it.

1. Header — 32 bytes, the whole of meta.zr.

• magic (ZREC) and version are ordinary fields, so opening a directory that is not a Zrecord store fails immediately instead of decoding garbage.
• codec is the store's one compression method, stamped at creation and immutable — there is no per-record tag anywhere.
• length (u64) is the total number of records | tail_chunk/tail_offset mark the last write position.
• There is no chunk count. Chunks are created in order and the frontier is always in the last one, so the store holds exactly chunks 0..=tail_chunk — a count would be a second copy of that fact to keep in sync.

const Codec = enum(u8) { raw = 0, zstd = 1, zstdict = 2, _ };
const Header = packed struct {
    magic: u32, version: u8, codec: Codec,
    tail_chunk: u32, tail_offset: u32, length: u64,
    _reserved: u80,
};

2. Record table — a .loc segment is 2^28 entries of 16 bytes (4 GiB, sparse), indexed directly. Mapping an index to a physical address is what makes random access efficient.

• chunk_id is the containing chunk | offset is the position within it | phys_length/logic_length are the stored and original sizes. The codec is not here: it is the header's, so a record is stored exactly the way the store is declared.

const RecordLoc = extern struct {
    offset: u32, phys_length: u32, logic_length: u32,
    chunk_id: u32,
};

There is no liveness flag. Every entry below length is live, because deletion swaps the tail into the hole rather than tombstoning.

3. No maximum length. The table grows a .loc segment at a time and the data grows a chunk at a time, so there is no static record-count cap to size against. The real bounds are the field widths — 2^32 chunks of 2^32 bytes (2^64 bytes total), 2^31 (2 GiB) per record — and disk.

Executor

1. Write. Writes are append-only; everything else is offset redirection.

• Append: compress concurrently → assign physical locations serially → flush concurrently → commit metadata. Data is durable-ordered before the record table, and the table before length, so a crash truncates rather than corrupts. A record never straddles two chunks; one that would not fit rolls over to a fresh chunk.
• Flush granularity is the store's, not the caller's. The batch size a client passes to append bounds memory only; zrecord's flush stage packs a shard's contiguous records into pwritevs capped at 256 KiB (flush_bytes). This decouples syscall size from batch and record size — a huge batch and a tiny one write identically, which matters because per-syscall writes above a few hundred KiB land on a slow regime on some filesystems/hosts (~1.4 GiB/s vs ~2.1 GiB/s at 256 KiB writes on the reference box). On POSIX the gather is a real pwritev with up to IOV_MAX (1024) iovecs. Windows has no vectored file write — WriteFileGather is async-only, OVERLAPPED, page-aligned — so it falls back to one positional write per record (correct, if not gathered). If Windows grows a synchronous vectored/IOCP file-write story, this is the one place writes there can be pulled up to parity; the byte-budget grouping is already in place, only the syscall differs.
• Delete: swap the last table entry into the deleted slot and drop the length by one. A batch is applied in descending index order, so each swap pulls from a slot no later target refers to. The index space stays dense — which is what the sampler needs, since it draws uniformly from 0..N and would otherwise keep hitting holes. The deleted record's bytes become garbage.

2. Read. Fill the destination memory concurrently, in place from the Python side (executed on async threads).

• Committed records are immutable, so the read path is lock free; length is published to readers through an atomic.
• Every record is read at the offset its table entry records — the record table is addressed by pure arithmetic, so random access is one pread for the location and one for the bytes, with no batching assumptions about layout. Locations are read once per batch (contiguous runs in one pread). Compressed records are read into a per-shard staging buffer and decompressed in place into the destination.

3. Concurrency model. Io.Group.async shards work by CPU count, and shards beyond the limit run inline on the calling thread.

• Shards receive contiguous blocks rather than a strided subset, keeping each worker's reads and writes sequential.
• Each shard creates one zstd context (ZSTD_CCtx to write, ZSTD_DCtx to read) and reuses it across every record it handles, rather than paying that setup per record. The dictionary (ZSTD_CDict/ZSTD_DDict) is immutable, so all shards share one, lock-free.
• Decompression writes straight into the caller's destination buffer, so there is no intermediate copy.

4. Garbage collection. Deletion leaves the record's bytes stranded, so space is reclaimed by an offline compact that rewrites the live records in place.

• Records are visited in physical order — which after swap-last deletes no longer matches table order — and repacked densely in that same order. A record therefore never moves to a higher address than it already had.
• Two things follow. Writing a record can never land on the bytes of a record not yet moved, so the rewrite is safe in place with no scratch copy of the store. And each table entry can be updated the instant its record lands, so the store is consistent at every point: an interrupted compaction leaves some records moved and the rest where they were, and re-running finishes the job.
• Records that are already in the right place are skipped, so a store with a small amount of garbage near the end is cheap to compact.
• Chunk files past the new frontier are closed and deleted.
• stats() reports live_bytes against chunk_bytes so callers can decide when it is worth running. Note the difference is an upper bound: a record never straddles a chunk boundary, so up to one record's worth per chunk is slack that compaction cannot remove.

5. File access.

• Metadata: meta.zr (32 bytes) plus one file per .loc segment, each 4 GiB (sparse), opened when the segment is created.
• Chunk data: 4 GiB files created at init, accessed concurrently through readPositionalAll/writePositionalAll.

Concurrency contract. gather and append are safe to call concurrently from many threads. delete and compact mutate the table in ways a lock-free reader would observe half-applied, so they require exclusive access to the store.