Loaderx

A compact, high-performance persistent record store with zero-copy batch gathering and transparent per-record compression, designed for single-machine AI training and serving pipelines

Zrecord is the typed persistent store; Loaderx is the sampler and prefetch loader that consumes Zrecord streams. They currently ship together while both layers mature, but their public responsibilities remain separate.

pip install loaderx

Wheels are published for Linux (glibc ≥ 2.17 and musl, x86-64 and arm64), macOS (≥ 11.0, Intel and Apple Silicon) and Windows (x64 and arm64). The bindings use cffi in ABI mode, so nothing links against the CPython ABI and one wheel per platform serves every supported Python.

Design Philosophy

loaderx is built around several core principles:

1. A pragmatic approach that prioritizes minimal memory overhead and minimal dependencies.
2. A strong focus on single-machine training workflows.
3. We implement based on NumPy semantics, persisted by the private native store engine.
4. An immortal (endless) step-based data loader, rather than the traditional epoch-based design—better aligned with modern ML training practices.
5. Dense and ragged are separate contracts, and the loader serves both. A dense stream stacks into one array per batch; a ragged one comes back as a list. Neither is padded, and equal length is never treated as a special case of variable length.

设计文档

• 性能优化与工程决策记录
• 分布式架构草案

Quick Start

from loaderx.zrecord import DenseStore
from loaderx.dataloader import DataLoader

data = np.load('data.npy', mmap_mode='r')
label = np.load('label.npy', mmap_mode='r')
with DenseStore.create('train_data', data.dtype, data.shape[1:]) as ds:
    ds.append(data)
with DenseStore.create('train_label', label.dtype, label.shape[1:]) as ds:
    ds.append(label)

loader = DataLoader({'data': DenseStore.open('train_data'),
                     'label': DenseStore.open('train_label')},
                    transform=lambda batch: batch)

for i, batch in enumerate(loader):
    if i >= 256:
        break

print(batch['data'].shape)
print(batch['label'].shape)

A batch is a dict {name: values}: each value is the stacked (batch_size, *item_shape) array for that stream. Every stream is gathered at the same indices, so record i lines up across them. The transform callback is the collate step — reshape, cast, stack — where values is the plain dense batch ready for the model.

Creating a dense store

import numpy as np
from loaderx.zrecord import DenseStore

data = np.load('data.npy', mmap_mode='r')
with DenseStore.create('train_data', data.dtype, data.shape[1:]) as ds:
    ds.append(data)

One record per slice along axis 0; a 1-D array (the usual shape of a label set) becomes a store of scalar records. The caller controls each append batch and can slice a large or mmapped array to set its own memory bound. Creation and opening are explicit: create requires a new path, while open requires an existing store. Open the result with DenseStore.open:

ds = DenseStore.open('train_data')

Python persists the exact type identity in store.zr: dtype plus the dense item_shape, or only dtype for ragged stores. It accepts no user metadata. Python selects the geometry from that schema and gives the private native engine only the runtime record boundaries it needs. Each Ragged record carries its shape in an inline little-endian u64 prefix. The native owners embed a shared physical record engine; that engine is an implementation detail. Raw bytes and pre-encoded images use a RaggedStore.create(path, dtype=np.uint8) rather than a second public storage API.

Records

One persistent format, two native execution contracts. DenseStore is the dense contract where every record is exactly one row of the recorded item_shape; reads are fixed-stride gathers and the batch shape follows from the type identity, so no per-record metadata is touched:

import numpy as np
from loaderx.zrecord import DenseStore

data = np.arange(64, dtype=np.float32).reshape(8, 2, 4)
with DenseStore.create('data', data.dtype, data.shape[1:]) as ds:
    ds.append(data)
ds = DenseStore.open('data')
ds[0, 5, 2]                          # (3, 2, 4) — shape from the persisted identity

A store is a collection of records, not an ndarray, so ds[0, 5, 2] is a record set — never ds[0][5][2]. A scalar selects one record; negatives wrap from the end. The ragged example below reads the same way.

RaggedStore is the ragged contract for variable-length records. It is a separate contract: :class:RaggedStore hands back a list of arrays, so a loader never has to carry row_splits around. dtype is unified and explicit; each record keeps its own shape, recorded per record as it is written and restored exactly on read — so records may differ in shape arbitrarily, and nothing is ever inferred from the source (an iterator can't tell you what its later records look like). Scalar shape == () is preserved; zero-byte arrays are rejected because physical records are nonempty. Densifying a list into a dense batch is the model's call — a plain numpy loop, wherever you need it:

from loaderx.zrecord import RaggedStore

seqs = [np.arange(L, dtype=np.int32) for L in (3, 1, 4, 1, 5)]
with RaggedStore.create('tokens', np.int32) as rs:
    rs.append(seqs)                       # dtype explicit; each record keeps its shape
rs = RaggedStore.open('tokens')

records = rs[0, 2, 4]                # list of ndarray — one per record, exact shapes
lengths = np.array([len(r) for r in records])
padded = np.zeros((len(records), lengths.max()), dtype=records[0].dtype)
for i, r in enumerate(records):
    padded[i, :len(r)] = r             # (B, max_len) — your policy, your loop

A DataLoader dynamically composes a dict of dense and ragged streams. Collation is the transform — a batch dict in, a batch dict out:

def collate(batch):
    return {'input_ids': batch['tokens'], 'label': batch['label']}

loader = DataLoader({'tokens': dense_tokens, 'label': labelset}, batch_size=32,
                    transform=collate)

Writable stores

The Store classes are loaderx's public persistence API: they are read-write, and append is explicit — one container of records, one batch, one native call. Nothing is buffered, inferred, or compressed for you.

from loaderx.zrecord import DenseStore, RaggedStore

ds = DenseStore.create('mnist/x', dtype=np.uint8, item_shape=(28, 28))  # identity committed now
ds.append(images[i:i + 1024])        # (B, 28, 28) → one batch, returns first index
ds.append(single_image[None])        # one sample is batch_size 1 — add the axis yourself
ds[:4]                               # read path is unchanged

tok = RaggedStore.create('tokens', dtype=np.int32)  # each record keeps its own shape
tok.append([seq_a, seq_b, seq_c])    # list in, list out — `tok[0, 2]` returns a list

The exact type identity is declared at creation, encoded and persisted by Python as msgpack. Dense identity contains only dtype and item_shape; ragged identity contains only dtype. Unexpected fields are rejected. append validates dtype and shape in Python, while native DenseStore independently enforces the persisted byte width. Dictionary training stays explicit and separate; creating a zstd_dict store requires the completed dictionary, so no valid store is published without one.

The store's maintenance pass-throughs are on the same object, normalized like reads (scalar, slice, array, bool mask):

ds.delete(np.arange(0, len(ds), 2))   # swap-last: index space stays dense, survivors reindex
ds.stats()                            # {'records', 'live_bytes', 'chunk_bytes', 'reclaimable'}
ds.compact()                          # reclaim the deleted bytes, in place — offline only

Deletion and compaction move records, so a multi-stream index you keep yourself must tolerate it — the guarantee is only that the live records are exactly 0..len(ds).

Codec notes

"zstd" compresses each record independently with plain zstd (level 3). Use it for general-purpose compression — it is fast and the default.

"zstd_dict" trains a shared dictionary on a sample of the data before writing any record, then compresses every record against it at level 19. The dictionary captures structure shared across records that per-record compression cannot see — a large win for many small, similar records (image tiles, token sequences).

The dictionary's cost is a cache footprint: every record's decompression references the shared dictionary window, so a larger dictionary means more cache misses on gather — the path a loader pays forever. Three named tiers (loaderx.zrecord.DICT_TIERS) preset the whole tradeoff — the dictionary size and how much data trains it — so a caller picks a tier, never a number:

The sample is the byte budget the dictionary trains on (a strided subset of the records), so each tier costs the same training time whatever the record size. At realistic image scale (768 KiB records) the tiers converge — on the earlier measurement box's structured data "balanced" and "max" both gather ~1.6 GiB/s at a 1.74x ratio — because a dictionary is a small fraction of a large frame. The tiers still matter at small record sizes, where the dict is most of a record and "max" trades gather throughput for ratio.

"zstd_dict" records can only be read from a store that has the dictionary (dict.zr). The dictionary is loaded on open and shared, lock-free, across all reader threads.

A dictionary must train on the settled, complete data. :func:train_dict is the standalone, manual training step — a numpy array, an iterable of records, or a typed store all train the same way, sized by a tier — and its bytes are handed to a store-writing path via dict_bytes. zstd_dict never trains by itself: a write without a dictionary is an error. A stream cannot train its own dictionary, but it can write with one trained on the settled data:

from loaderx.utils import train_dict
from loaderx.zrecord import DenseStore, RaggedStore

# train once on the settled data — a standalone, reusable artifact
d = train_dict(settled_array, tier="balanced")

# then any new store can install it and append explicitly
with RaggedStore.create('tokens', np.int32, codec='zstd_dict', dict_bytes=d) as ds:
    ds.append(token_generator)
with DenseStore.create('data', data.dtype, data.shape[1:],
                       codec='zstd_dict', dict_bytes=d) as ds:
    ds.append(data)

Changing an existing store's codec is a rewrite, not a store mutation: read the records as numpy values and append them to a new store with the new codec. A whole-store slice preserves index order, which keeps multi-stream alignment. There is no dedicated recode or store-to-store path because the ordinary read and append contracts already express the operation:

from loaderx.utils import train_dict
from loaderx.zrecord import DenseStore

with DenseStore.open("src") as s, \
     DenseStore.create("dst", dtype=s.dtype, item_shape=s.item_shape,
                       codec="zstd_dict", dict_bytes=train_dict(s)) as d:
    d.append(s[:])                             # ndarray -> dense append

RaggedStore is the same shape: s[:] returns list[np.ndarray], which is exactly its append input, and dtype is all the destination needs. The append pipeline bounds its own compression working memory; there is no public chunk parameter. dst must not already hold a store.

Important: Do not use "zstd_dict" while data is still changing (through deletes or compaction below the Python layer). The dictionary captures a snapshot of the data; training it before the data settles wastes compression. Train the dictionary once preprocessing is complete and the content is final.

Multi-stream stores

A zrecord store is one stream; a training sample is usually several named streams (skeleton + label + id, tokens + label, ...). Composition is a plain Python dict passed to DataLoader. There is no persistent wrapper, manifest, directory convention or bundle mutation API. DataLoader verifies that all streams have the same length, then gathers every stream with the same indices.

from loaderx.zrecord import DenseStore, RaggedStore
from loaderx.dataloader import DataLoader

root = "xsub/train"
with DenseStore.create(root + "/joint", joint.dtype, joint.shape[1:]) as s:
    s.append(joint)
with DenseStore.create(root + "/label", label.dtype, label.shape[1:]) as s:
    s.append(label)
with RaggedStore.create(root + "/token", np.int32) as s:
    s.append(seqs)

streams = {
    "joint": DenseStore.open(root + "/joint"),
    "label": DenseStore.open(root + "/label"),
    "token": RaggedStore.open(root + "/token"),
}
streams["joint"][0, 5, 2]            # each stream keeps its own index API
loader = DataLoader(streams, batch_size=256)
batch = next(loader)                      # {name: values}, index-aligned

Equal-length and variable-length are both just stores — DenseStore (one fixed-shape record per sample) and RaggedStore (variable row count per record). The loader does not interpret either: it fetches by index and packs a dict, so a dense stream's batch value is the stacked (B, *item_shape) array and a ragged stream's is a list of per-record arrays. No padding is imposed — densify to a fixed shape however the model needs (a plain numpy loop), or reshape/stack in the transform collate.

CPU → GPU transfer

loaderx hands over CPU batches; getting them to the accelerator is the transform's job — the one place your framework is already imported. The batch dict is a plain {name: numpy array}, zero-copy on the way out, so a device transfer is one call per stream:

import torch

device = "cuda:0"
def to_device(batch):
    return {k: torch.from_numpy(v).to(device, non_blocking=True)
            for k, v in batch.items()}

loader = DataLoader(streams, transform=to_device)
for batch in loader:
    model(batch)                        # already on device

A non_blocking=True copy is genuinely asynchronous only when its source is pinned. loaderx does not pin memory for you — pinning is framework-owned (torch's .pin_memory(), CUDA's cudaHostAlloc), and a vendor-free core stops exactly at the CPU batch. Pin in the transform what you copy:

def to_device(batch):
    return {k: torch.from_numpy(v).pin_memory().to(device, non_blocking=True)
            for k, v in batch.items()}

JAX is the same shape — jax.device_put is already an asynchronous handoff on GPU:

import jax

def to_device(batch):
    return {k: jax.device_put(v) for k, v in batch.items()}

The transfer runs on the transform stage and never touches loaderx internals: the copy overlaps the next batch's gather/transform, and any pinned pool is the caller's to own and reuse. This is the entire H2D answer — there is no pin= hook or device backend, because the only unified thing a multi-framework loader can own is the CPU batch.

For practical integration examples, please refer to the Data2Latent repository

Benchmarks

Dense and Ragged are measured separately because they expose different contracts, but every store table uses the same columns. scripts/bench_dense.py measures fixed-shape random gather, scripts/bench_ragged.py measures variable-shape records, and scripts/bench.py covers machine, sampler, and the end-to-end loader comparison. Every path runs through the public Python binding, so CFFI, NumPy allocation, and Ragged list/shape reconstruction are timed.

Methodology

The 1.8.5 snapshots below are each one qualitative pass on a warm page cache. Within one workload every backend receives identical source records and random index plans. Before timing, the benchmark sweeps every record and validates every planned result for exact dtype, shape, order, and values. logical write and logical gather divide uncompressed NumPy payload bytes by elapsed time; they measure bytes accepted or returned by the public API, not physical storage bandwidth. Write is page-cache ingestion with durability deferred consistently across backends; call sync() explicitly when measuring Zrecord durability. Disk size is allocated blocks, not sparse apparent size. krecords/s is gather record throughput and p95 is the 95th-percentile latency of one random gather batch. Results are comparable within one workload table, not across payload distributions or geometries. Every backend name states its actual codec; the full default set is required rather than silently skipped when a package is missing. Dense and Ragged use the same CHW RGB image generator, record count, batch plan and seed. Dense fixes every image at (3, 224, 224); Ragged changes only H and W.

Machine — one local workstation (AMD Ryzen AI 9 HX PRO 370, 12 cores / 24 threads):

The benchmark process sees all 24 threads and is not memory-limited by cgroup. The 16 GiB shared-memory mount accommodates the four-worker, 64 MiB-batch torch pipeline. Store reads run on the ordinary page cache.

Large Vision Records

Fixed-Shape Dense

Zrecord against array-store alternatives: random batch gather, 2,500 CHW RGB records, batch 256. Every image is (3, 224, 224) and comes from the same spatial model used by the variable-shape benchmark.

Fixed-resolution vision records — 147 KiB per record, 36.8 MiB per batch:

At 147 KiB per record, Zrecord-raw reaches 9.4 GiB/s and is 2.3x npy-mmap-raw; plain zstd gathers at 7.2 GiB/s while reducing the corpus 14.68x. LMDB and Arrow IPC are competitive raw record stores, while codecs tied to whole IPC batches or Parquet row groups pay read amplification on random gathers. DenseStore demonstrates that typed record ownership and per-record compression do not turn fixed tensors into an object-store slow path.

Variable-Shape Ragged

This workload contains 2,500 variable-resolution CHW RGB images. Height and width are independently lognormal and clipped to 64..512 (observed medians 223 and 224), totaling 417.1 MiB of logical uint8 payload. Each of 50 random batches contains 256 records. Every backend persists payload plus exact shape and must return an ordered list[np.ndarray] of (3, H, W) arrays; a flat byte list or a one-dimensional variable-length abstraction is not enough.

Zrecord-raw is the fastest raw record path, with LMDB close behind; Arrow IPC is the strongest raw typed-file alternative. Zrecord-zstd delivers 5.0 GiB/s of logical payload while reducing the corpus to 26.9 MiB. HDF5, Arrow IPC, Parquet, ArrayRecord and TileDB show the same framework/codec tradeoffs in both tables; compressed batch, chunk and row-group formats pay read amplification on random records.

The shared generator makes compression ratios directly comparable across contracts: Zrecord zstd is 14.68x Dense versus 15.52x Ragged, and zstdict is 22.74x versus 23.57x. The remaining difference comes from the H/W distribution and Ragged shape metadata, not a different image entropy model.

Small Token Records

Both token workloads come from the same real corpus: WikiText-103 raw train, tokenized with GPT-2 and stored as int32 IDs. Preparation is outside every measurement. scripts/prepare_tokens.py preserves nonempty text boundaries, combines fragments shorter than 16 tokens, and splits records at 512 tokens into tokens.npy plus offsets.npy; both benchmark scripts mmap those files.

Fixed Token Blocks

The Dense workload ignores text boundaries and packs the stream into 200,000 fixed int32[512] records: 2 KiB per record and 390.6 MiB logical payload. Each of 100 random batches gathers 256 records.

The contiguous NumPy baseline is strongest when the whole corpus is one fixed typed matrix. Zrecord-raw reaches 2.74 Mrecords/s while retaining independent record semantics; per-record zstd halves disk and still returns 0.79–0.89 Mrecords/s. LMDB's B-tree/page overhead is visible in both throughput and disk.

Variable Token Sequences

The Ragged workload keeps 200,000 real text records of 16..512 tokens: p10 28, median 129, mean 138.8, p90 254, totaling 105.9 MiB. It uses the same 256-record, 100-batch random plan and every backend must return ordered list[np.ndarray] with exact int32 values and original one-dimensional shapes.

Here the record contract, not bulk byte bandwidth, is the useful scale. Zrecord's three codecs return 302–345 krecords/s with approximately 1 ms p95; the dictionary gives the best disk ratio without reducing throughput relative to plain zstd.

Sampler

Index generation on its own, IID (with replacement), 1M index space, against NumPy's modern API. The µs-scale figures fluctuate with box load; the stable signal is the approximately 1.9x margin, roughly flat across batch sizes.

End-to-End DataLoader

The original full input pipeline comparison (sample, fetch, collate, hand over a batch), same workload, 4 transform workers, batch 256 (64 MiB). The peak RSS column (added with the uniform-codec rewrite) is the highest resident set size of the whole process tree while batches are flowing. These are retained published loader results; current defaults keep the very slow Grain setup optional through --only grain.

storage is the actual backing store used by each pipeline. This is an end-to-end systems comparison, not a scheduler-only comparison over one shared storage layer: Torch reads read-only NumPy mmap files, Loaderx reads Zrecord, and Grain reads ArrayRecord.

At 64 MiB per batch the loader is DRAM-bound, not sampler-bound — the per-batch gather is the whole story, and the transform threads keep it at store-gather speed while the Python side collates. The memory is the prefetch buffers plus the two Store handles (raw + label stores over the same backing data). loaderx prefetches in threads inside one process, so workers share one interpreter, one numpy and one set of gather buffers; torch and grain run a worker process per prefetch thread, which is most of their RSS (torch's peak also includes the shared-memory collated batches). Compressed loaderx is slightly ahead of torch; loaderx-raw is 2.0x faster than torch and 8.1x faster than grain.

Free-threaded Python. loaderx targets free-threaded builds (no GIL), and the key sections are also measured on 3.14t — the question is whether anything gains when nothing is GIL-limited. These are opt-in runs (a second interpreter and --transform augment); they are not part of the default all:

• Sampler — zsampler draws ~1.9–2x faster than free-threaded numpy (the 256-row run was a noisy 2.46x), the same broad margin as on the GIL build: a batch is one Zig call either way.

• Historical Store snapshot — these older rows predate the symmetric benchmark matrix and are retained only for the GIL/free-threaded comparison. The free-threaded numbers track the GIL table closely. zrecord-raw gathers 8.9 vs 8.8 GiB/s and zrecord-zstd 4.7 vs 4.6 GiB/s. The no-GIL table was run without blosc2 so its import could not change the process mode. blosc2 was measured separately at 678 MiB/s; its extension emits a warning and re-enables the GIL, so that cell is not a no-GIL result. ArrayRecord has no usable 3.14t extension. Random gather, 256 KiB records:

  store	CPython 3.14 (GIL)	free-threaded 3.14t
  zrecord-raw	8803 MiB/s	8878 MiB/s
  zrecord-zstd	4603 MiB/s	4707 MiB/s
  npy-mmap-raw	4408 MiB/s	3974 MiB/s
  hdf5-raw	2288 MiB/s	2210 MiB/s
  blosc2-zstd	646 MiB/s	678 MiB/s (GIL enabled)
  tensorstore-zarr3-gzip	299 MiB/s	323 MiB/s

• Loader, identity — unchanged on both interpreters at 64 MiB per batch:

  loader	CPython 3.14 (GIL)	free-threaded 3.14t
  loaderx	72.2 batches/s	71.1 batches/s
  loaderx-raw	136.7 batches/s	136.0 batches/s

• Loader, CPU-heavy transform — the one place the free-threaded build matters. A transform runs on the transform threads, so the GIL serializes it on standard CPython — the case where worker processes win, and the reason for the free-threaded build. --transform augment (a Python-bound per-sample loop), 4 workers. Torch and Grain have no free-threaded rows, so their GIL worker processes are shown only as context:

  loader	CPython 3.14 (GIL)	free-threaded 3.14t
  loaderx	33.1 batches/s	41.9 batches/s
  loaderx-raw	36.4 batches/s	51.9 batches/s
  torch	35.9 batches/s	—
  grain	10.2 batches/s	—

  Free-threaded wins because the transform parallelizes across the prefetch threads: 1.27x on loaderx and 1.43x on raw at 256 KiB. The margin is bounded by the 64 MiB gather, which remains DRAM-bandwidth-heavy.

Conclusion — why the numbers look like this.

Every hot path is batched natively. Zsampler draws a whole batch of indices; DenseStore gathers and decompresses a whole fixed-shape batch in one CFFI call; RaggedStore gathers each shape-prefixed record once before Python validates its prefix and reconstructs the exact arrays. The speedup is not bought with sampling shortcuts: the IID draw is unbiased like NumPy's (Lemire with rejection, so uniformity costs nothing over a real index space).

The layouts match what a training loader does. Zrecord is built for random record access: DenseStore gathers fixed-width records directly into one ndarray; RaggedStore restores independently shaped records from inline shape/payload entries. Array stores are built primarily for contiguous scans, so a scattered batch fights their layout. A dense raw gather fans out across every core where NumPy fancy indexing is one thread; RaggedStore instead trades some raw specialization for compression and a complete variable-shape persistence model.

Compression is in the storage kernel, and there is one codec. zrecord-zstd is not "storage plus a codec": the layout, the multi-core decompress and the GIL-free copy are one path, so turning compression on costs part of a margin, not an order of magnitude. The ratio is the data, not the format: these samples are barely compressible, and on a smooth image set plain zstd reaches 7.6x and the dictionary (at the "max" tier) 16.5x.

The loader gap is architecture, not storage. loaderx uses threads and never ends an epoch, so a step pays no IPC and never waits on an epoch boundary; torch restarts per epoch with worker processes. Here compressed loaderx is 1.1x faster than torch and 4.3x faster than grain; raw loaderx is 2.0x and 8.1x faster, respectively. Storage also differs per loader — each reads from what it was built for — so the loader table is a different comparison from either store table, not a rerun. The one crack in the thread model is a CPU-heavy transform, which the GIL serializes — that is exactly what free-threaded Python removes, so loaderx is developed and benchmarked against free-threaded builds first.

What these numbers do not claim. Everything runs with a warm page cache: this measures the access path, not cold storage or disk. disk is allocated blocks; zrecord files grow to their written frontier, while TensorStore's sharding carries its own layout overhead. write is page-cache ingestion with durability deferred, matching the other backends — zrecord's own durability (sync/close) is a separate, explicit cost that none of the store tables pay. This is a single qualitative pass — the µs-scale sampler timings and the loader batches/s fluctuate with box load (on these 12 cores the compressed loader trails the raw one substantially), so treat the absolute numbers as ballpark and the cross-backend margins as the signal.

Reproduction

python3 scripts/bench_dense.py
python3 scripts/bench_ragged.py

python3 scripts/prepare_tokens.py wiki.train.tokens /tmp/wikitext-gpt2
python3 scripts/bench_dense.py --workload tokens \
  --token-corpus /tmp/wikitext-gpt2 --records 200000 --batches 100
python3 scripts/bench_ragged.py --workload tokens \
  --token-corpus /tmp/wikitext-gpt2 --records 200000 --batches 100

prepare_tokens.py also accepts Hugging Face WikiText Parquet shards directly. The published run used Salesforce/wikitext, config wikitext-103-raw-v1, revision refs/convert/parquet, train shards 0000.parquet then 0001.parquet; their SHA-256 values are respectively 74da360f23826045b3e6ac6375411fdb15f003030aa74f2596ed08b857cb9212 and ba090ac30dbf5461e8dcbdd1a1b8e6f3cf9c2c756d64f0c1220450acd514f720. The focused token defaults omit formats that are already represented in the larger vision matrix; --only can select any registered backend explicitly.

Real-data verification: NTU RGB-D skeletons

The vision tables above are synthetic; the token tables use real WikiText-103. As a separate historical ground-truth check, loaderx was run end to end on NTU RGB-D skeleton data — 114,480 raw .skeleton files, 120 action classes, 25 joints — processed into the ST-GCN N C T V M layout (per-sample (3, 300, 25, 2) float32) for the xsub/xview protocols. The .npy outputs of the standard preprocessing pipeline were treated as ground truth. This verification was not rerun with the synthetic benchmarks above; its throughput is retained as a separate historical 12-core result.

Correctness — the read path is bit-exact against the ground truth:

• Full scan of all 228,356 records (joint float32 + label int64, all four splits) through DenseStore: byte-for-byte identical to the reference npy.
• A DataLoader over joint + label + an index stream, run under all three sampler modes (sequential, iid, cyclic): every received batch is bit-exact to the ground truth at its own declared indices, and the streams stay index-aligned.
• Sampler semantics hold on real index spaces: sequential walks in order, cyclic draws a full cycle without replacement, iid is deterministic per seed.

Storage — zstd on this data:

Sizes above are for one split (xview/val, 38,132 records); across all four splits the zstd joint stores total 4.97 GB against 41 GB of raw npy (~8x).

Throughput (180 KB per record, warm page cache, 12 physical cores):

zstd decompression reads ~8x fewer bytes than raw storage, so the compressed store gathers faster than the raw one (zstd 4587 MiB/s vs raw 1792 MiB/s on the same split).

The npy intermediate is optional. Parse the skeleton files in parallel and feed each fixed-shape ndarray produced by the parser directly to DenseStore.append. This writes the store in one pass with no npy staging or second read; zrecord bounds its compression working memory independently.

Current Limitations

• Single-host only; multi-host training is not supported.
• A single sample must be at most 2 GiB (2^31 bytes). There is no fixed record count: length is a u64 and the record table grows on demand, so how many records a store holds is bounded by its total chunk capacity (up to 2^64 bytes) divided by the average record size — e.g. roughly 2^44 records at 1 MiB each, 2^33 at 2 GiB each.
• Metadata is read and written as the host's struct layout, so a store carries the host's byte order and is not portable to a machine of the opposite endianness. Every published platform is little-endian, so this only matters if you build for one yourself.

Build

zig build                       # host shared objects, into loaderx/lib/
zig build test                  # native store suite, in both Debug and ReleaseFast
python3 scripts/test_loaderx.py # Python integration suite against the real build
uv pip install -r scripts/requirements-bench.txt
python3 scripts/bench_dense.py  # fixed-shape store comparison
python3 scripts/bench_ragged.py # variable-length store comparison
python3 scripts/bench.py        # machine, sampler, and dense loader layers

The Zig side is tested for behaviour only; throughput is measured from Python, through the binding a client actually uses. Optional benchmark contenders are skipped when not installed. See Benchmarks.

Publishing

Zig cross-compiles every target from one machine, so releases need no CI matrix:

zig build dist                    # every platform, into zig-out/dist/<wheel tag>/
python3 scripts/build_wheels.py   # one wheel per platform, plus the sdist

The dist directories are named after their Python wheel platform tag, so the tag mapping lives in exactly one place (dist_targets in build.zig). glibc and macOS minimums are pinned in the target triple, which is what makes manylinux_2_17 and macosx_11_0 honest rather than aspirational. Each wheel is checked after packing: it must carry this platform's libraries and no others.

The sdist ships sources only. Installing from it runs zig build through setup.py, so it needs the Zig compiler; wheel users never hit that path.

Zsampler

Index Generator: a high-performance sampler implemented in Zig. Every mode is a pure function of (seed, step), so a run resumes exactly by seeking to a step — there is no epoch to track, in keeping with the endless step-based loader.

from loaderx.zsampler import Sampler

sampler = Sampler(1_000_000, 256, Sampler.Mode.IID, seed=42)
indices = sampler.next()

1. Sequential — traverse the index space in order through a fixed-size sliding window, treating the space as a circular queue so the tail never truncates.
2. IID — draw each index uniformly at random with replacement. Unbiased (Lemire with rejection), matching NumPy. Simplest, but coverage is uneven over any short run.
3. Cyclic — without replacement, round-robin. Each cycle traverses a fresh permutation of the whole index space, so within a cycle every record appears exactly once and no batch repeats an index — coverage is even by construction, which keeps how often each sample is seen uniform. The permutation is a stateless bijection (a small Feistel network over the index space, brought into range by cycle-walking), so a million-record shuffle materializes nothing the size of the dataset and reshuffling each cycle is free. Every batch is exactly batch_size: an endless step-based loader has no final partial batch to special-case, so when batch_size does not divide the length the cycle's remainder is dropped — a different remainder each cycle, since the permutation changes, so every record is still reached over time.

Zrecord

Zrecord is loaderx's typed persistent record store. Its private native runtime is the byte-oriented engine beneath the public DenseStore and RaggedStore contracts.

The private Python CFFI surface is kept together in loaderx/_store.py because both handles share one libstore, error model and lifecycle. Python owns store.zr and all dtype/shape semantics. In Zig, src/store/dense.zig and src/store/ragged.zig adapt trusted runtime record boundaries to the shared physical engine, while src/store.zig is the sole C ABI export/composition root. zrecord.py remains the unified Python-facing API.

Trust boundary. Python and Zig are one zrecord implementation, not two independently supported products. loaderx/_store.py, the C ABI in src/store.zig, and the native handles are private implementation details; no compatibility or defensive-validation contract is provided to code that calls them directly. Public inputs are normalized and validated once, in whichever half can express the rule most simply, and the internal Python→CFFI→Zig call then trusts that contract instead of repeating it at every layer. This is not a license to trust storage or the operating system: native code still validates persisted addresses and lengths, buffer bounds and integer overflow, short or failed I/O, codec output, locking, and commit ordering. Those checks protect normal crash/corruption handling and native memory safety; checks that only defend against bypassing the public Python API do not belong in zrecord.

1. RecordEngine is an unordered physical store made of N records. Records are independent and carry no ordering, so every index and slice operation is equivalent to a gather.
2. It hands the Store layer a dense index space: the live records are always exactly 0..N. Deletion preserves that by swapping the tail into the hole, which means an index is stable only until something is deleted. Named streams are composed dynamically by a plain Python dict; DataLoader validates that the independent stores have equal lengths.
3. The engine reads and writes byte ranges. Python owns type identity and selects Dense or Ragged geometry; the native owners receive only the runtime byte geometry and own physical transaction semantics. Dense stores persist one fixed-width physical record per logical record. Ragged stores persist one variable-width physical record per logical record: [u64le ndim][u64le dims...][payload]. Shape and payload therefore share one location, codec frame, append commit, and swap-delete operation.
4. The IO model (append | read | delete) is batch-oriented and shape-agnostic. Store owners expose record boundaries through a compile-time source interface; the engine has one append operation and carries no Dense, Ragged, dtype, or array-shape semantics. A single-record operation is just the batch_size == 1 case.
5. The engine owns its temporary memory internally — allocation and release are explicit.
6. A store has exactly one codec in its native physical header, fixed at creation and immutable afterwards. Every record is compressed and decompressed independently:

| tag   |  name     | algorithm                              |
|-------|-----------|----------------------------------------|
|   0   |  raw      | none                                   |
|   1   |  zstd     | zstd (plain, level 3)                  |
|   2   |  zstdict  | zstd with a trained dictionary (level 19) |

7. Compression is transparent to the client:
   • Compression runs concurrently across all cores. A compressed store never falls back to raw: each record is stored as the codec's output, even when an incompressible record's frame is larger than its input — write raw if the data does not compress.
   • Decompression writes straight into the caller's destination memory (gather), with no intermediate buffer and no extra copy.
   • zstd is the one transparent codec — faster than Deflate at both ends and a better ratio, so there is no reason to carry a second. It is vendored C, built for every platform by Zig, so the one-wheel-per-platform story is unchanged.
   • zstd_dict additionally trains one dictionary on a sample of the data (stored as dict.zr) and compresses every record against it. Because each record is still independent, random access is unchanged — but the dictionary carries the structure shared across records, which per-record compression cannot see. On many small, similar records (image tiles, token sequences) this is a large win: a smooth-image set that plain zstd takes to 7.6x compresses 16.5x with a large dictionary. The dictionary is loaded once on open and shared, lock-free, across all reader threads. The dictionary size is chosen from the DICT_TIERS presets (see Codec notes).
   • A zstd_dict store needs its dictionary to read every record; a raw or zstd store rejects an unexpected dictionary as malformed state.

Persistence format

Native storage is metadata plus chunked data. The extension is the type: .zr files are store-global singletons, .loc files are record-table segments, .chunk files are record data:

store/
  ├── store.zr     exact Python msgpack type identity
  ├── meta.zr      header (global state)
  ├── dict.zr      zstd dictionary (only in dict stores)
  ├── 0.loc        record table segment [0, 2^28)
  ├── 1.loc        record table segment [2^28, 2^29)
  ├── 0.chunk      record data
  └── 1.chunk

Metadata (store.zr + meta.zr + {id}.loc)

Files are read and written positionally — pread/pwrite at computed offsets, no mmap. The header is a naturally aligned extern struct and a .loc segment an array of 16-byte RecordLocs, exactly as wide as they declare, so a location is one pread/pwrite of 16 bytes at a computed offset and there is no serializer anywhere in the code. Active header fields are contiguous; reserved bytes exist only at the end of its fixed 32-byte footprint.

The record table is partitioned into {id}.loc segments so it can grow by appending a segment instead of reserving the maximum. The id→segment mapping is pure arithmetic — seg = idx >> 28, off = (idx & (2^28−1)) × 16 — so a segment needs no per-record bookkeeping. Committed entries are immutable; one shared fd-table lock keeps the in-memory ArrayList stable while readers index it during the rare append that adds another segment.

1. Python type identity — store.zr is exactly one immutable msgpack object. Dense stores contain only dtype and item_shape; Ragged stores contain only dtype. Python writes it only after the engine and required dictionary are durable, and open first acquires the native lifetime lock before reading and validating it. Dense record width is derived once from dtype/item_shape and passed to the native handle as runtime geometry; it is not independently persisted as a second authority. There is no format version or legacy kind dispatch.

2. Physical header — 32 bytes, the whole of meta.zr.

• codec is the store's one compression method, stamped at creation and immutable — there is no per-record tag anywhere.
• length (u64) is the physical record count; it equals logical length for both dense stores and inline ragged stores. tail_chunk/tail_offset mark the last write position.
• There is no chunk count. Chunks are created in order and the frontier is always in the last one, so the store holds exactly chunks 0..=tail_chunk — a count would be a second copy of that fact to keep in sync.

const Codec = enum(u8) { raw = 0, zstd = 1, zstdict = 2, _ };
const Header = extern struct {
    length: u64,
    tail_chunk: u32,
    tail_offset: u32,
    codec: Codec,
    reserved: [15]u8,
};

3. Record table — a .loc segment addresses up to 2^28 entries of 16 bytes (4 GiB), indexed directly. The file starts empty and grows as contiguous loc runs are written; 4 GiB is its address boundary, not its initial file size. Mapping an index to a physical address is what makes random access efficient.

• chunk_id is the containing chunk | offset is the position within it | phys_length/logic_length are the stored and original sizes. The codec is not here: it is the header's, so a record is stored exactly the way the store is declared.

const RecordLoc = extern struct {
    offset: u32, phys_length: u32, logic_length: u32,
    chunk_id: u32,
};

There is no liveness flag. Every entry below length is live, because deletion swaps the tail into the hole rather than tombstoning.

4. No maximum length. The table grows a .loc segment at a time and the data grows a chunk at a time, so there is no static record-count cap to size against. The real bounds are the field widths — 2^32 chunks of 2^32 bytes (2^64 bytes total), 2^31 (2 GiB) per record — and disk.

Executor

1. Write. Writes are append-only; everything else is offset redirection. The codec is immutable store state, so append and gather dispatch once at their entry points into separate raw, zstd, or zstd-dictionary implementations. Their contexts and workers are deliberately not unified: only validation, location bounds, locking, and the final metadata transaction are shared. Geometry is equally explicit across the whole stack: Python derives Dense record width from its type identity and configures the fixed-stride handle, while RaggedStore supplies offsets for its shape-prefixed records. The owners turn those ABI inputs into compile-time record sources and destinations; the engine has one append and one gather operation. There is no generic public handle or ABI.

• Compressed append: workers claim individual records, compress into independent slots, reserve physical offsets in completion order through a short frontier lock, and issue positional payload writes in parallel. Logical IDs remain in the loc table, so physical completion order does not change random gather. The caller waits for every payload before committing metadata. Append publishes through the page cache and is intentionally lazy; sync() makes the committed frontier durable in payload → record-table → header order and propagates any sync failure. A record never straddles two chunks; one that would not fit rolls over to a fresh chunk. Chunk files start empty and positional writes extend them to the current frontier; the 4 GiB u32 offset range is a logical capacity, not a sparse preallocation requirement. Compression fans out to the executor's CPU count without a separate append-specific worker cap. Each producer configures its CCtx or shared immutable CDict once, then starts every independent record frame with ZSTD_compress2.
• Raw append preserves the stronger invariant already supplied by Python: every record in one Dense or packed Ragged batch is a boundary inside one contiguous source buffer. After locating the records, the engine consumes that buffer in plan order and passes each contiguous chunk run directly to writePositionalAll. A chunk file boundary is the only split; there is no per-record iovec construction, byte-budget flush, payload copy, or platform-specific syscall path. Zig's std.Io handles short writes and maps the same positional operation to POSIX and Windows implementations.
• Delete: swap the last table entry into the deleted slot and drop the length by one. A batch is applied in descending index order, so each swap pulls from a slot no later target refers to. The index space stays dense — which is what the sampler needs, since it draws uniformly from 0..N and would otherwise keep hitting holes. The deleted record's bytes become garbage.

2. Read. Fill the destination memory concurrently, in place from the Python side (executed on async threads).

• Committed records are immutable and length is published through an atomic. A gather takes one shared fd-table lock so a rare chunk/segment ArrayList growth cannot invalidate its file handles; record I/O itself stays lock free.
• Every record is read at the offset its table entry records — the record table is addressed by pure arithmetic, so random access is one pread for the location and one for the bytes, with no batching assumptions about layout. Each shard reads one location and immediately reads/decompresses that record; there is no separate metadata phase or sequential-run special case. Compressed records use a per-shard staging buffer and decode in place into the destination.

3. Concurrency model. Io.Group.async shards work by CPU count, and shards beyond the limit run inline on the calling thread.

• Shards receive contiguous blocks rather than a strided subset, keeping each worker's reads and writes sequential.
• Each shard creates one zstd context (ZSTD_CCtx to write, ZSTD_DCtx to read) and reuses it across every record it handles, rather than paying that setup per record. The dictionary (ZSTD_CDict/ZSTD_DDict) is immutable, so all shards share one, lock-free.
• Decompression writes straight into the caller's destination buffer, so there is no intermediate copy.

4. Garbage collection. Deletion leaves the record's bytes stranded, so space is reclaimed by an offline compact that rewrites the live records in place.

• Records are visited in physical order — which after swap-last deletes no longer matches table order — and repacked densely in that same order. A record therefore never moves to a higher address than it already had.
• Two things follow. Writing a record can never land on the bytes of a record not yet moved, so the rewrite is safe in place with no scratch copy of the store. And each table entry can be updated the instant its record lands, so the store is consistent at every point: an interrupted compaction leaves some records moved and the rest where they were, and re-running finishes the job.
• Records that are already in the right place are skipped, so a store with a small amount of garbage near the end is cheap to compact.
• Chunk files past the new frontier are closed and deleted; the final retained chunk is truncated to its new tail offset, releasing its old physical tail.
• Delete has already made the live loc table a dense 0..length prefix, so compact also deletes loc segments beyond that prefix and truncates its final segment to exactly length * 16 bytes.
• stats() reports live_bytes against chunk_bytes so callers can decide when it is worth running. Note the difference is an upper bound: a record never straddles a chunk boundary, so up to one record's worth per chunk is slack that compaction cannot remove.

5. File access.

• Metadata: meta.zr (32 bytes) plus naturally growing .loc segments, each with a 4 GiB maximum address range.
• Chunk data: naturally growing files with a 4 GiB logical capacity, accessed concurrently through readPositionalAll/writePositionalAll. No path depends on filesystem sparse-file support.

Concurrency contract. One native handle owns a store at a time through a lifetime, nonblocking exclusive lock on meta.zr; a second handle or process fails with StoreBusy. Within that handle, gather and append are safe to call concurrently from many threads. Append remains single-writer; only rare chunk/segment fd-table growth briefly waits for active gathers. delete and compact mutate the table in ways a reader would observe half-applied, so they require exclusive access to the store.