Loaderx

Zrecord is a rebuildable high-performance on-disk record container built from authoritative source data and scripts. create is append-only, close publishes the container Header, and open is read-only. To change content, rebuild it at a new path.

Zrecord is the typed on-disk container; Loaderx is the sampler and prefetch loader that consumes Zrecord streams. They currently ship together while both layers mature, but their public responsibilities remain separate.

pip install loaderx

Wheels are published for Linux (glibc ≥ 2.17 and musl, x86-64 and arm64), macOS (≥ 11.0, Intel and Apple Silicon) and Windows (x64 and arm64). The bindings use cffi in ABI mode, so nothing links against the CPython ABI and one wheel per platform serves every supported Python.

Design Philosophy

loaderx is built around several core principles:

1. A pragmatic approach that prioritizes minimal memory overhead and minimal dependencies.
2. A strong focus on single-machine training workflows.
3. We implement based on NumPy semantics, persisted by the private native store engine.
4. An immortal (endless) step-based data loader, rather than the traditional epoch-based design—better aligned with modern ML training practices.
5. Dense and ragged are separate contracts, and the loader serves both. A dense stream stacks into one array per batch; a ragged one comes back as a list. Neither is padded, and equal length is never treated as a special case of variable length.

设计文档

• 性能优化与工程决策记录
• 分布式架构草案

Quick Start

import numpy as np

from loaderx.zrecord import Dense
from loaderx.dataloader import DataLoader

data = np.load('data.npy', mmap_mode='r')
label = np.load('label.npy', mmap_mode='r')
with Dense.create('train_data', data.dtype, data.shape[1:]) as ds:
    ds.append(data)
with Dense.create('train_label', label.dtype, label.shape[1:]) as ds:
    ds.append(label)

data_store = Dense.open('train_data')
label_store = Dense.open('train_label')
loader = DataLoader({'data': data_store, 'label': label_store},
                    transform=lambda batch: batch)
for i, batch in enumerate(loader):
    if i >= 256:
        break

print(batch['data'].shape)
print(batch['label'].shape)

loader.close()
data_store.close()
label_store.close()

A batch is a dict {name: values}: each value is the stacked (batch_size, *item_shape) array for that stream. Every stream is gathered at the same indices, so record i lines up across them. The transform callback is the collate step — reshape, cast, stack — where values is the plain dense batch ready for the model.

Creating a dense store

import numpy as np
from loaderx.zrecord import Dense

data = np.load('data.npy', mmap_mode='r')
with Dense.create('train_data', data.dtype, data.shape[1:]) as ds:
    ds.append(data)

One record per slice along axis 0; a 1-D array (the usual shape of a label set) becomes a store of scalar records. The caller controls each append batch and can slice a large or mmapped array to set its own memory bound. Creation and opening are explicit: create requires a new path and returns an append-only writer; leaving its context calls close(), which publishes the result. open requires an existing container produced by a successful build and returns a read-only reader. Open the result with Dense.open:

ds = Dense.open('train_data')
batch = ds[:]
ds.close()

Python defines the exact record schema: dtype plus the dense item_shape, or only dtype for ragged stores. The MsgPack bytes live opaquely in the static page at the front of meta.zr; Zig persists them but never interprets them. The schema accepts no user metadata. Python selects the geometry and gives the private native engine only the runtime record boundaries it needs. Each Ragged record carries a u8 rank followed by inline little-endian u64 dimensions. One native physical engine consumes the trusted Dense stride or Ragged offsets. Append inputs are strictly NumPy arrays: Dense takes one batched ndarray and Ragged takes an iterable of ndarrays. Raw bytes and pre-encoded images are made explicit with np.frombuffer(raw, dtype=np.uint8) and stored in a Ragged.create(path, dtype=np.uint8) rather than creating a second public storage API.

Records

One persistent format, two native execution contracts. Dense is the dense contract where every record is exactly one row of the recorded item_shape; reads are fixed-stride gathers and the batch shape follows from the schema, so no per-record metadata is touched:

import numpy as np
from loaderx.zrecord import Dense

data = np.arange(64, dtype=np.float32).reshape(8, 2, 4)
with Dense.create('data', data.dtype, data.shape[1:]) as ds:
    ds.append(data)
ds = Dense.open('data')
ds[0, 5, 2]                          # (3, 2, 4) — shape from the persisted schema
ds.close()

A store is a collection of records, not an ndarray, so ds[0, 5, 2] is a record set — never ds[0][5][2]. A scalar selects one record; indices must be in 0..len(ds)-1. The ragged example below reads the same way.

Ragged is the ragged contract for variable-length records. It is a separate contract: :class:Ragged hands back a list of arrays, so a loader never has to carry row_splits around. dtype is unified and explicit; each record keeps its own shape, recorded per record as it is written and restored exactly on read — so records may differ in shape arbitrarily, and nothing is ever inferred from the source (an iterator can't tell you what its later records look like). Scalar shape == () is preserved; zero-byte arrays are rejected because physical records are nonempty. Densifying a list into a dense batch is the model's call — a plain numpy loop, wherever you need it:

from loaderx.zrecord import Ragged

seqs = [np.arange(L, dtype=np.int32) for L in (3, 1, 4, 1, 5)]
with Ragged.create('tokens', np.int32) as rs:
    rs.append(seqs)                       # dtype explicit; each record keeps its shape
rs = Ragged.open('tokens')

records = rs[0, 2, 4]                # list of ndarray — one per record, exact shapes
rs.close()
lengths = np.array([len(r) for r in records])
padded = np.zeros((len(records), lengths.max()), dtype=records[0].dtype)
for i, r in enumerate(records):
    padded[i, :len(r)] = r             # (B, max_len) — your policy, your loop

A DataLoader dynamically composes a dict of dense and ragged streams. Collation is the transform — a batch dict in, a batch dict out:

def collate(batch):
    return {'input_ids': batch['tokens'], 'label': batch['label']}

loader = DataLoader({'tokens': dense_tokens, 'label': labelset}, batch_size=32,
                    transform=collate)
batch = next(loader)
loader.close()

The transform runs once for each gathered batch on a loader transform worker. Dense values are independent writable contiguous arrays; Ragged records share one backing allocation per batch. The return value is handed to the consumer unchanged. Calls may run concurrently and complete out of order, so the callback must be thread-safe; exceptions are propagated to the consumer. Keep shared mutable state and nested parallel runtimes out of the callback.

Numba can optionally accelerate a CPU-heavy Dense transform while releasing the GIL. Compile it before timing, then call it from the ordinary transform:

import numba
import numpy as np

@numba.njit(nogil=True, parallel=False)
def normalize_u8(x):
    out = np.empty(x.shape, dtype=np.float32)
    for i in range(x.size):
        out.flat[i] = x.flat[i] / 255.0
    return out

normalize_u8(np.zeros((1, 3, 224, 224), dtype=np.uint8))  # compile warmup

def transform(batch):
    batch["image"] = normalize_u8(batch["image"])
    return batch

Numba is neither a loaderx dependency nor a loader backend. Its compilation warmup belongs outside build or loader benchmark timing.

Creating containers

Dense.create and Ragged.create return append-only writers. append is explicit — one container of records becomes one native batch. Dense append is synchronous and borrows an already-contiguous ndarray without a snapshot copy. Ragged append consumes its iterable once into one owned packed buffer, then completes the native append before returning. Nothing is inferred.

from loaderx.zrecord import Dense, Ragged

ds = Dense.create('mnist/x', dtype=np.uint8, item_shape=(28, 28))
ds.append(images[i:i + 1024])        # synchronous native batch; returns None
ds.append(single_image[None])        # one sample is batch_size 1 — add the axis yourself
ds.close()                            # publish before opening

tok = Ragged.create('tokens', dtype=np.int32)
tok.append([seq_a, seq_b, seq_c])
tok.close()

with Dense.open('mnist/x') as ds:
    first_four = ds[:4]               # opened containers are read-only

Dense and Ragged append both report native errors in the current call and return only after accepting the batch. A writer cannot be read, and a reader cannot be appended to. close() on a writer publishes the container Header; close() on a reader releases it. Dense, Ragged, and DataLoader support with for scoped lifetimes. Content is never changed in place: rerun the authoritative build at a new path, validate it, then switch consumers to it.

The exact schema is declared at creation and encoded by Python as MsgPack. Dense schema contains only dtype and item_shape; Ragged schema contains only dtype. Structured, subarray, object, and metadata-bearing dtypes are not supported: their semantics do not round-trip through one canonical NumPy dtype string. The encoded schema has 4064 bytes available in the fixed 4096-byte metadata page. Ragged schema size is effectively fixed; Dense schema size grows only with the integer item_shape, so the physical limit is far above any practical NumPy array rank. Unexpected fields are rejected. append validates dtype and shape in Python, then passes the derived byte width to the private Dense operation. It does not coerce Python lists, tuples, bytes, bytearrays, or other array-like objects; callers convert them with np.asarray or np.frombuffer first. Dictionary training stays explicit and separate; creating a zstd_dict store requires the completed dictionary, so no valid store is published without one.

Codec notes

"zstd" compresses each record independently with plain zstd (level 3). Use it for general-purpose compression — it is fast and the default.

"zstd_dict" trains a shared dictionary on a sample of the data before writing any record, then compresses every record against it at level 19. The dictionary captures structure shared across records that per-record compression cannot see — a large win for many small, similar records (image tiles, token sequences).

The dictionary's cost is a cache footprint: every record's decompression references the shared dictionary window, so a larger dictionary means more cache misses on gather — the path a loader pays forever. Three named tiers (loaderx.zrecord.DICT_TIERS) preset the whole tradeoff — the dictionary size and how much data trains it — so a caller picks a tier, never a number:

The sample is the byte budget the dictionary trains on (a strided subset of the records), so each tier costs the same training time whatever the record size. At realistic image scale (768 KiB records) the tiers converge — on the earlier measurement box's structured data "balanced" and "max" both gather ~1.6 GiB/s at a 1.74x ratio — because a dictionary is a small fraction of a large frame. The tiers still matter at small record sizes, where the dict is most of a record and "max" trades gather throughput for ratio.

"zstd_dict" records can only be read from a store that has the dictionary (dict.zr). The dictionary is loaded on open and shared, lock-free, across all reader threads.

A dictionary must train on the settled, complete data. :func:train_dict is the standalone, manual training step — a numpy array, an iterable of records, or a typed container all train the same way, sized by a tier — and its bytes are handed to a container-writing path via dict_bytes. zstd_dict never trains by itself: a write without a dictionary is an error. A stream cannot train its own dictionary, but it can write with one trained on the settled data:

from loaderx.utils import train_dict
from loaderx.zrecord import Dense, Ragged

# train once on the settled data — a standalone, reusable artifact
d = train_dict(settled_array, tier="balanced")

# then any new store can install it and append explicitly
with Ragged.create('tokens', np.int32, codec='zstd_dict', dict_bytes=d) as ds:
    ds.append(token_generator)
with Dense.create('data', data.dtype, data.shape[1:],
                       codec='zstd_dict', dict_bytes=d) as ds:
    ds.append(data)

Changing an existing store's codec is a rewrite, not a store mutation: read the records as numpy values and append them to a new store with the new codec. A whole-store slice preserves index order, which keeps multi-stream alignment. There is no dedicated recode or store-to-store path because the ordinary read and append contracts already express the operation:

from loaderx.utils import train_dict
from loaderx.zrecord import Dense

with Dense.open("src") as s, \
     Dense.create("dst", dtype=s.dtype, item_shape=s.item_shape,
                       codec="zstd_dict", dict_bytes=train_dict(s)) as d:
    d.append(s[:])                             # ndarray -> dense append

Ragged is the same shape: s[:] returns list[np.ndarray], which is exactly its append input, and dtype is all the destination needs. The native compression path bounds its own working memory; there is no public chunk parameter. dst must not already hold a store.

Important: Train the dictionary from settled authoritative input before building the container. Training it before preprocessing is complete wastes compression and does not describe the final records.

Multi-stream stores

A zrecord store is one stream; a training sample is usually several named streams (skeleton + label + id, tokens + label, ...). Composition is a plain Python dict passed to DataLoader. There is no persistent wrapper, manifest, directory convention or bundle mutation API. DataLoader verifies that all streams have the same length, then gathers every stream with the same indices.

from loaderx.zrecord import Dense, Ragged
from loaderx.dataloader import DataLoader

root = "xsub/train"
with Dense.create(root + "/joint", joint.dtype, joint.shape[1:]) as s:
    s.append(joint)
with Dense.create(root + "/label", label.dtype, label.shape[1:]) as s:
    s.append(label)
with Ragged.create(root + "/token", np.int32) as s:
    s.append(seqs)

streams = {
    "joint": Dense.open(root + "/joint"),
    "label": Dense.open(root + "/label"),
    "token": Ragged.open(root + "/token"),
}
streams["joint"][0, 5, 2]            # each stream keeps its own index API
loader = DataLoader(streams, batch_size=256)
batch = next(loader)                  # {name: values}, index-aligned

loader.close()
for stream in streams.values():
    stream.close()

Equal-length and variable-length are both just stores — Dense (one fixed-shape record per sample) and Ragged (variable row count per record). The loader does not interpret either: it fetches by index and packs a dict, so a dense stream's batch value is the stacked (B, *item_shape) array and a ragged stream's is a list of per-record arrays. No padding is imposed — densify to a fixed shape however the model needs (a plain numpy loop), or reshape/stack in the transform collate.

CPU → GPU transfer

loaderx hands over CPU batches; getting them to the accelerator is the transform's job — the one place your framework is already imported. The batch dict is a plain {name: numpy array}, zero-copy on the way out, so a device transfer is one call per stream:

import torch

device = "cuda:0"
def to_device(batch):
    return {k: torch.from_numpy(v).to(device, non_blocking=True)
            for k, v in batch.items()}

streams = {
    "joint": Dense.open(root + "/joint"),
    "label": Dense.open(root + "/label"),
    "token": Ragged.open(root + "/token"),
}
loader = DataLoader(streams, transform=to_device)
for batch in loader:
    model(batch)                        # already on device

Call loader.close() when the training loop exits. The streams remain caller-owned and should be closed at the application lifecycle boundary.

A non_blocking=True copy is genuinely asynchronous only when its source is pinned. loaderx does not pin memory for you — pinning is framework-owned (torch's .pin_memory(), CUDA's cudaHostAlloc), and a vendor-free core stops exactly at the CPU batch. Pin in the transform what you copy:

def to_device(batch):
    return {k: torch.from_numpy(v).pin_memory().to(device, non_blocking=True)
            for k, v in batch.items()}

JAX is the same shape — jax.device_put is already an asynchronous handoff on GPU:

import jax

def to_device(batch):
    return {k: jax.device_put(v) for k, v in batch.items()}

The transfer runs on the transform stage and never touches loaderx internals: the copy overlaps the next batch's gather/transform, and any pinned pool is the caller's to own and reuse. This is the entire H2D answer — there is no pin= hook or device backend, because the only unified thing a multi-framework loader can own is the CPU batch.

For practical integration examples, please refer to the Data2Latent repository

Benchmarks

Dense and Ragged are measured separately because they expose different contracts, but every store table uses the same columns. scripts/bench_dense.py measures fixed-shape random gather, scripts/bench_ragged.py measures variable-shape records, and scripts/bench.py covers machine, sampler, and the end-to-end loader comparison. Every path runs through the public Python binding, so CFFI, NumPy allocation, and Ragged list/shape reconstruction are timed.

Methodology

The results below are one complete run from the same checkout on a warm page cache. They are not three-run medians: an unexpected result is traced separately instead of being hidden by repeated aggregation. Within each store workload every backend receives identical source records. Correctness and timing use independent deterministic zsampler IID streams. Loader backends receive the same source and seed but use their own shipped samplers, so their exact permutations differ. Before gather timing, the store benchmark sweeps every record and validates a fixed number of IID batches for exact dtype, shape, order, and values. logical write and logical gather divide uncompressed NumPy payload bytes by elapsed time; they measure bytes accepted or returned by the public API, not physical storage bandwidth. Writable containers are created before timing. logical write starts when the already-generated source enters the backend, includes byte encoding, packing, key construction and Arrow array construction, and ends after logical commit/finalization returns. No backend requests fsync, LMDB env.sync(), or another stable-media durability operation. Reusable source preparation is outside that timer; in particular, zstd_dict trains its standalone dictionary first.

The measured Store paths are explicit:

write: make_data returns -> writer setup [untimed]
       -> start -> encode/pack -> append/write -> logical finalize -> return -> stop
       -> resource-only cleanup [untimed]

read:  open -> full warm sweep -> IID correctness stream(seed) [untimed]
       -> IID timing stream(seed + 1): sampler.next() [untimed]
       -> start one gather -> public return -> stop
       -> destroy returned batch [untimed]
       -> repeat timed calls until their accumulated time is at least 2 seconds

Logical finalize means Zrecord Header publication, LMDB transaction commit, or an Arrow/Parquet footer; none of these paths requests stable-media sync. Disk size is allocated blocks, not sparse apparent size. krecords/s is gather record throughput and p95 is the 95th-percentile latency of one random gather batch. Results are comparable within one workload table, not across payload distributions or geometries. The finalized output is opened read-only before timing. After the warm sweep and correctness stream, an independent IID stream draws fresh indices until timed gather calls accumulate at least two seconds. IID sampling is uniform with replacement, and sampler time is excluded. Output allocation, reads, decompression and reconstruction are included, while open, close and destruction after return are not. Every backend name states its actual codec; the full default set is required rather than silently skipped when a package is missing. Dense and Ragged use the same CHW RGB image generator, record count, batch plan and seed. Dense fixes every image at (3, 224, 224); Ragged changes only H and W.

Machine — one local workstation (AMD Ryzen AI 9 HX PRO 370, 12 cores / 24 threads):

The benchmark process sees all 24 threads and is not memory-limited by cgroup. The 16 GiB shared-memory mount accommodates the four-worker, 36.8 MiB-batch torch pipeline. Store reads run on the ordinary page cache.

Large Vision Records

Fixed-Shape Dense

Zrecord against array-store alternatives: random batch gather, 2,500 CHW RGB records, batch 256. Every image is (3, 224, 224) and comes from the same spatial model used by the variable-shape benchmark.

Fixed-resolution vision records — 147 KiB per record, 36.8 MiB per batch:

At 147 KiB per record, Zrecord-raw reaches 15.5 GiB/s and is 3.4x npy-mmap-raw; plain zstd gathers at 11.5 GiB/s while reducing the corpus 14.69x. LMDB and Arrow IPC are competitive raw record stores, while codecs tied to whole IPC batches or Parquet row groups pay read amplification on random gathers. Dense demonstrates that typed record ownership and per-record compression do not turn fixed tensors into an object-store slow path.

Variable-Shape Ragged

This workload contains 2,500 variable-resolution CHW RGB images. Height and width are independently lognormal and clipped to 64..512 (observed medians 223 and 224), totaling 417.1 MiB of logical uint8 payload. Each of 50 random batches contains 256 records. Every backend persists payload plus exact shape and must return an ordered list[np.ndarray] of (3, H, W) arrays; a flat byte list or a one-dimensional variable-length abstraction is not enough.

Zrecord-raw is 1.8x LMDB and 3.2x Arrow IPC in logical gather. Zrecord-zstd delivers 9.8 GiB/s of logical payload while reducing the corpus to 26.9 MiB. HDF5, Arrow IPC, Parquet, ArrayRecord and TileDB show the same framework/codec tradeoffs in both tables; compressed batch, chunk and row-group formats pay read amplification on random records.

The shared generator makes compression ratios directly comparable across contracts: Zrecord zstd is 14.69x Dense versus 15.52x Ragged, and zstdict is 22.74x versus 23.58x. The remaining difference comes from the H/W distribution and Ragged shape metadata, not a different image entropy model.

Small Token Records

Both token workloads come from the same real corpus: WikiText-103 raw train, tokenized with GPT-2 and stored as int32 IDs. Preparation is outside every measurement. scripts/prepare_tokens.py preserves nonempty text boundaries, combines fragments shorter than 16 tokens, and splits records at 512 tokens into tokens.npy plus offsets.npy; both benchmark scripts mmap those files.

Fixed Token Blocks

The Dense workload ignores text boundaries and packs the stream into 200,000 fixed int32[512] records: 2 KiB per record and 390.6 MiB logical payload. Each IID batch gathers 256 records; fresh draws continue until timed gathers accumulate at least two seconds.

The contiguous NumPy baseline is strongest when the whole corpus is one fixed typed matrix. Zrecord-raw reaches 3.94 Mrecords/s while retaining independent record semantics; the per-record zstd codecs halve disk and still return 1.05–1.23 Mrecords/s. LMDB's B-tree/page overhead is visible in both throughput and disk.

Variable Token Sequences

The Ragged workload keeps 200,000 real text records of 16..512 tokens: p10 28, median 129, mean 138.8, p90 254, totaling 105.9 MiB. It uses the same 256-record, random plan and every backend must return ordered list[np.ndarray] with exact int32 values and original one-dimensional shapes.

Here the record contract, not bulk byte bandwidth, is the useful scale. Zrecord's three codecs return 550–666 krecords/s with less than 0.6 ms p95; the dictionary gives the best disk ratio and is slightly ahead of plain zstd in this pass.

Sampler

Index generation on its own, IID (with replacement), 1M index space, against NumPy's modern API. The µs-scale figures fluctuate with box load; this pass shows an approximately 2.1–2.3x margin across batch sizes.

End-to-End DataLoader

The full input pipeline comparison (sample, fetch, collate, hand over a batch) uses exactly the Dense vision source above: 2,500 (3,224,224) uint8 records from make_vision_records, 4 high-level workers, and batch 256 (36.8 MiB). After warmup, throughput and memory are collected during one 200-batch qualitative pass. peak PSS sums proportional set size across the process tree, apportioning mapped shared and copy-on-write pages instead of counting each once per worker. It does not include ordinary kernel page-cache pages used by pread, while resident mmap pages are attributed to the mapping process, so it is a process-mapping diagnostic rather than total pipeline physical memory. aggregate RSS deliberately sums each process's full resident set: on Linux it double-counts shared/COW pages, which explains process-tree RSS inflation but is neither physical memory nor a projection of Windows committed memory. The explicit spawn row is the relevant no-fork control; Windows itself still requires a native run. Torch fork is kept because it is the normal Linux mode, while spawn exposes the ownership model used on platforms without fork. Grain setup remains optional through --only grain; the published command selects it explicitly in the same workload matrix.

storage is the actual backing store used by each pipeline. This is an end-to-end systems comparison, not a scheduler-only comparison over one shared storage layer: Torch reads read-only NumPy mmap files, Loaderx reads Zrecord, and Grain reads ArrayRecord.

At 36.8 MiB per batch the per-batch gather dominates the tiny sampler cost, and the transform threads overlap Python-side collation with the next gather. The memory is the source, Zrecord container and bounded in-flight batches. loaderx prefetches in threads inside one process, so workers share one interpreter, one NumPy runtime and one set of gather buffers. With source geometry and entropy held constant, raw is 1.43x compressed loaderx; compressed loaderx is 1.47x Torch fork, 1.38x Torch spawn and 3.65x Grain, while raw is 2.10x, 1.97x and 5.22x faster. Torch's aggregate RSS is high because Linux fork mappings are counted repeatedly; it is not a total-memory ratio against Zrecord's unaccounted page cache. The explicit torch-spawn row removes fork/COW dependence. Because this Dataset keeps only mmap paths, spawn does not copy the full corpus into every worker; a Windows Dataset holding Python lists or in-memory arrays would be a different, deliberately harsher workload.

The Torch-only worker sweep runs each count once. Worker 0 is an in-process baseline (145.5–153.8 batches/s and about 1485 MiB peak PSS in the two equivalent rows), so the process-context comparison starts at one worker:

Spawn peak PSS grows from 1891 to 4218 MiB as workers rise from one to eight, while fork grows from 1678 to 2016 MiB because it retains COW sharing. At eight workers spawn uses 2.09x fork's peak PSS and throughput has already flattened. This demonstrates no-fork memory pressure; it is not labeled OOM because this 31 GiB machine completed the run. Fork aggregate RSS grows faster because Linux counts shared/COW mappings in every process, so RSS is diagnostic rather than physical memory.

CPU-heavy transform solutions. The same Python per-sample transform exposes the GIL bottleneck on standard CPython. Numba is an explicit solution, not the default: --transform numba-nogil compiles the equivalent batch transform with nogil=True. The other explicit solution runs the Python transform on free-threaded CPython. Numba compilation is warmed before timing. Store microbenchmarks are not repeated under free-threaded Python because Loaderx has no separate no-GIL Store implementation; the Store table above applies to both.

Peak PSS for compressed/raw was 1204/1189 MiB with GIL Python, 1290/1262 MiB with Numba, and 1190/1190 MiB with free-threaded Python. These are two deployment solutions to the transform bottleneck, not claims that Store itself was optimized for either runtime.

Conclusion — why the numbers look like this.

Every hot path is batched natively. Zsampler draws a whole batch of indices; Dense gathers and decompresses a whole fixed-shape batch in one CFFI call; Ragged gathers each shape-prefixed record once before Python validates its prefix and reconstructs the exact arrays. The speedup is not bought with sampling shortcuts: the IID draw is unbiased like NumPy's (Lemire with rejection, so uniformity costs nothing over a real index space).

The layouts match what a training loader does. Zrecord is built for random record access: Dense gathers fixed-width records directly into one ndarray; Ragged restores independently shaped records from inline shape/payload entries. Array stores are built primarily for contiguous scans, so a scattered batch fights their layout. A dense raw gather fans out across the shared Executor budget where NumPy fancy indexing is one thread; Ragged instead trades some raw specialization for compression and a complete variable-shape persistence model.

Compression is in the storage kernel, and there is one codec. zrecord-zstd is not "storage plus a codec": the layout, the multi-core decompress and the GIL-free copy are one path, so turning compression on costs part of a margin, not an order of magnitude. The ratio is the data, not the format: in the current Dense structured-vision workload, plain zstd reaches 14.69x and the balanced dictionary reaches 22.74x.

Loader results combine architecture and storage. loaderx uses threads and never ends an epoch, so a step pays no IPC and never waits on an epoch boundary; torch uses finite shuffled epochs, worker processes and shared-memory handoff. Here compressed loaderx is 1.47x Torch fork, 1.38x Torch spawn and 3.65x Grain; raw loaderx is 2.10x, 1.97x and 5.22x faster, respectively. Storage also differs per loader — each reads from what it was built for — so the loader table is a different comparison from either store table, not a rerun. The one crack in the thread model is a CPU-heavy Python transform, which the GIL serializes. Numba nogil=True and free-threaded Python are measured as two explicit solutions rather than silently changing the default transform.

What these numbers do not claim. Everything runs with a warm page cache: this measures the access path, not cold storage or disk. disk is allocated blocks, and zrecord files grow to their written frontier. logical write measures source adaptation through logical finalization after writer setup; it does not benchmark durability, stronger transactional guarantees, or reusable dictionary training. Arrow IPC and Parquet use 256-record groups, so random batches pay their real group-level read amplification. Ragged Zrecord arrays are views into one batch allocation, while most byte-store adapters return independent copies. This is a single benchmark run, while each Store read path accumulates at least two timed seconds — the µs-scale sampler timings and the loader batches/s fluctuate with box load (on these 12 cores the compressed loader trails the raw one by about 25%), so treat the absolute numbers as ballpark and the cross-backend margins as the signal.

Reproduction

.venv/bin/python scripts/bench_dense.py
.venv/bin/python scripts/bench_ragged.py

.venv/bin/python scripts/prepare_tokens.py wiki.train.tokens /tmp/wikitext-gpt2
.venv/bin/python scripts/bench_dense.py --workload tokens \
  --token-corpus /tmp/wikitext-gpt2 --records 200000 --batches 100
.venv/bin/python scripts/bench_ragged.py --workload tokens \
  --token-corpus /tmp/wikitext-gpt2 --records 200000 --batches 100

.venv/bin/python scripts/bench.py sampler
.venv/bin/python scripts/bench.py loader \
  --only loaderx,loaderx-raw,torch,torch-spawn,grain --workers 4
.venv/bin/python scripts/bench.py loader --only torch,torch-spawn \
  --workers 0,1,2,4,8
.venv/bin/python scripts/bench.py loader \
  --only loaderx,loaderx-raw --workers 4 --transform python-loop
.venv/bin/python scripts/bench.py loader --only loaderx,loaderx-raw \
  --workers 4 --transform numba-nogil

.venv-t/bin/python scripts/bench.py loader --only loaderx,loaderx-raw \
  --workers 4 --transform python-loop

prepare_tokens.py also accepts Hugging Face WikiText Parquet shards directly. The published run used Salesforce/wikitext, config wikitext-103-raw-v1, revision refs/convert/parquet, train shards 0000.parquet then 0001.parquet; their SHA-256 values are respectively 74da360f23826045b3e6ac6375411fdb15f003030aa74f2596ed08b857cb9212 and ba090ac30dbf5461e8dcbdd1a1b8e6f3cf9c2c756d64f0c1220450acd514f720. The focused token defaults omit formats that are already represented in the larger vision matrix; --only can select any registered backend explicitly. The loader rows used Numba 0.67.0, Torch 2.13.0 and Grain 0.2.18; all benchmark dependencies are pinned in scripts/requirements-bench.txt. Temporary stores used the ordinary disk-backed /tmp filesystem, not /dev/shm.

Real-data verification: NTU RGB-D skeletons

The vision tables above are synthetic; the token tables use real WikiText-103. As a separate historical ground-truth check, loaderx was run end to end on NTU RGB-D skeleton data — 114,480 raw .skeleton files, 120 action classes, 25 joints — processed into the ST-GCN N C T V M layout (per-sample (3, 300, 25, 2) float32) for the xsub/xview protocols. The .npy outputs of the standard preprocessing pipeline were treated as ground truth. This verification was not rerun with the synthetic benchmarks above; its throughput is retained as a separate historical 12-core result.

Correctness — the read path is bit-exact against the ground truth:

• Full scan of all 228,356 records (joint float32 + label int64, all four splits) through Dense: byte-for-byte identical to the reference npy.
• A DataLoader over joint + label + an index stream, run under all three sampler modes (sequential, iid, cyclic): every received batch is bit-exact to the ground truth at its own declared indices, and the streams stay index-aligned.
• Sampler semantics hold on real index spaces: sequential walks in order, cyclic draws a full cycle without replacement, iid is deterministic per seed.

Storage — zstd on this data:

Sizes above are for one split (xview/val, 38,132 records); across all four splits the zstd joint stores total 4.97 GB against 41 GB of raw npy (~8x).

Throughput (180 KB per record, warm page cache, 12 physical cores):

zstd decompression reads ~8x fewer bytes than raw storage, so the compressed store gathers faster than the raw one (zstd 4587 MiB/s vs raw 1792 MiB/s on the same split).

The npy intermediate is optional. Parse the skeleton files in parallel and feed each fixed-shape ndarray produced by the parser directly to Dense.append. This writes the store in one pass with no npy staging or second read; zrecord bounds its compression working memory independently.

Current Limitations

• Single-host only; multi-host training is not supported.
• A single sample must be at most 2 GiB (2^31 bytes). There is no fixed record count: length is a u64 and the record table grows on demand. Practical store size is bounded by disk and the platform's positional file-offset range.
• Metadata is read and written as the host's struct layout, so a store carries the host's byte order and is not portable to a machine of the opposite endianness. Every published platform is little-endian, so this only matters if you build for one yourself.

Build

zig build                       # host shared objects, into loaderx/lib/
zig build test                  # native store suite, in both Debug and ReleaseFast
python3 scripts/test_loaderx.py # Python integration suite against the real build
uv pip install --python .venv/bin/python -r scripts/requirements-bench.txt
.venv/bin/python scripts/bench_dense.py  # fixed-shape store comparison
.venv/bin/python scripts/bench_ragged.py # variable-length store comparison
.venv/bin/python scripts/bench.py        # machine, sampler, and dense loader layers

The Zig side is tested for behaviour only; throughput is measured from Python, through the binding a client actually uses. Optional benchmark contenders are skipped when not installed. See Benchmarks.

Publishing

Zig cross-compiles every target from one machine, so releases need no CI matrix:

zig build dist                    # every platform, into zig-out/dist/<wheel tag>/
python3 scripts/build_wheels.py   # one wheel per platform, plus the sdist

The dist directories are named after their Python wheel platform tag, so the tag mapping lives in exactly one place (dist_targets in build.zig). glibc and macOS minimums are pinned in the target triple, which is what makes manylinux_2_17 and macosx_11_0 honest rather than aspirational. Each wheel is checked after packing: it must carry this platform's libraries and no others.

The sdist ships sources only. Installing from it runs zig build through setup.py, so it needs the Zig compiler; wheel users never hit that path.

Zsampler

Index Generator: a high-performance sampler implemented in Zig. Every mode is a pure function of (seed, step), so a run resumes exactly by seeking to a step — there is no epoch to track, in keeping with the endless step-based loader.

from loaderx.zsampler import Sampler

sampler = Sampler(1_000_000, 256, Sampler.Mode.IID, seed=42)
indices = sampler.next()       # borrowed until this sampler's next draw
saved = indices.copy()         # retain across draws only when needed

next() and iteration return a view of one reusable uint64 batch buffer. The contents stay unchanged until the next explicit draw from that Sampler; copy only plans that must outlive it. DataLoader consumes each view synchronously before drawing again.

1. Sequential — traverse the index space in order through a fixed-size sliding window, treating the space as a circular queue so the tail never truncates.
2. IID — draw each index uniformly at random with replacement. Unbiased (Lemire with rejection), matching NumPy. Simplest, but coverage is uneven over any short run.
3. Cyclic — without replacement, round-robin. Each cycle traverses a fresh permutation of the whole index space, so within a cycle every record appears exactly once and no batch repeats an index — coverage is even by construction, which keeps how often each sample is seen uniform. The permutation is a stateless bijection (a small Feistel network over the index space, brought into range by cycle-walking), so a million-record shuffle materializes nothing the size of the dataset and reshuffling each cycle is free. Every batch is exactly batch_size: an endless step-based loader has no final partial batch to special-case, so when batch_size does not divide the length the cycle's remainder is dropped — a different remainder each cycle, since the permutation changes, so every record is still reached over time.

Zrecord

Zrecord is loaderx's rebuildable typed record container. Its private native runtime is the byte-oriented engine beneath the public Dense and Ragged contracts.

The private Python CFFI surface is kept together in loaderx/_store.py because both contracts share one libstore, error model, lifecycle and opaque Store handle. Python owns all schema semantics; Zig stores the opaque schema bytes in meta.zr. In Zig, src/store.zig adapts each Dense stride or Ragged offsets call to the shared physical engine and is the sole C ABI export/composition root. zrecord.py remains the unified Python-facing API.

Trust boundary. Python and Zig are one zrecord implementation, not two independently supported products. loaderx/_store.py, the C ABI in src/store.zig, and the native handles are private implementation details; no defensive-validation contract is provided to code that calls them directly. Public inputs are normalized and validated once, in whichever half can express the rule most simply, and the internal Python→CFFI→Zig call then trusts that contract instead of repeating it at every layer. Python owns the creator/reader capability model and chooses append, gather, creator Header sync, or reader handle release; the Zig engine stores no writable/reader mode. Native create/open still select read-write/exclusive or read-only/shared file handles, because those are OS access and locking mechanics rather than API capabilities. This is not a license to trust storage or the operating system: native code still validates persisted addresses and lengths, buffer bounds and integer overflow, short or failed I/O, codec output, locking, and commit ordering. Those checks protect normal I/O behavior, basic malformed-store rejection and native memory safety; checks that only defend against bypassing the public Python API do not belong in zrecord.

1. RecordEngine is an unordered physical store made of N records. Records are independent and carry no ordering, so every index and slice operation is equivalent to a gather.
2. It hands the container layer a dense index space: records are exactly 0..N. Named streams are composed dynamically by a plain Python dict; DataLoader validates that the independent containers have equal lengths.
3. The engine reads and writes byte ranges. Python owns the record schema and selects Dense or Ragged geometry; the private ABI receives only the runtime byte geometry. Dense stores persist one fixed-width physical record per logical record. Ragged stores persist one variable-width physical record per logical record: [u64le ndim][u64le dims...][payload]. Shape and payload therefore share one location, codec frame, and append publication.
4. The IO model (append | read) is batch-oriented and shape-agnostic. Per-call adapters expose record boundaries through a compile-time source interface; the engine has one append operation and carries no Dense, Ragged, dtype, or array-shape semantics. A single-record operation is just the batch_size == 1 case.
5. The engine owns its temporary memory internally — allocation and release are explicit.
6. A store has exactly one codec in its native physical header, fixed at creation and immutable afterwards. Every record is compressed and decompressed independently:

| tag   |  name     | algorithm                              |
|-------|-----------|----------------------------------------|
|   0   |  raw      | none                                   |
|   1   |  zstd     | zstd (plain, level 3)                  |
|   2   |  zstdict  | zstd with a trained dictionary (level 19) |

7. Compression is transparent to the client:
   • Compression runs concurrently across the shared Executor budget. A compressed store never falls back to raw: each record is stored as the codec's output, even when an incompressible record's frame is larger than its input — write raw if the data does not compress.
   • Decompression writes straight into the caller's destination memory (gather), with no intermediate buffer and no extra copy.
   • zstd is the one transparent codec — faster than Deflate at both ends and a better ratio, so there is no reason to carry a second. It is vendored C, built for every platform by Zig, so the one-wheel-per-platform story is unchanged.
   • zstd_dict additionally trains one dictionary on a sample of the data (stored as dict.zr) and compresses every record against it. Because each record is still independent, random access is unchanged — but the dictionary carries the structure shared across records, which per-record compression cannot see. On many small, similar records (image tiles, token sequences) this is a large win: the current Dense structured-vision set is 14.69x with plain zstd and 22.74x with the balanced dictionary. The dictionary is loaded once on open and shared, lock-free, across all reader threads. The dictionary size is chosen from the DICT_TIERS presets (see Codec notes).
   • A zstd_dict store needs its dictionary to read every record; a raw or zstd store rejects an unexpected dictionary as malformed state.

Persistence format

The 2.0 clean break does not change the current on-disk format. That format is the current implementation only: there is no compatibility layer, migration, version dispatch, checksum, or recovery facility. Zrecord is not the authority for irreplaceable data. Keep authoritative source data and reproducible build scripts; after an interrupted build, storage failure, implementation change, or content change, rebuild a complete container at a new path.

Native storage uses a fixed file set:

store/
  ├── meta.zr      4096-byte static Header/schema page + RecordLoc table
  ├── data.zr      payload stream
  └── dict.zr      zstd dictionary (only in dict stores)

Metadata (meta.zr)

Files are read and written positionally — pread/pwrite at computed offsets, no mmap. meta.zr starts with one fixed 4096-byte static page: a naturally aligned 32-byte Header, then the opaque MsgPack schema and unused zero padding. An array of 16-byte RecordLocs starts at offset 4096. Record i is one pread/pwrite at 4096 + i * 16; there is no variable table base, segment mapping, or rollover fd table.

1. Python schema — bytes 32..32+schema_length are exactly one immutable MsgPack object. Dense stores contain only dtype and item_shape; Ragged stores contain only dtype. Native create persists these bytes together with the physical container but does not decode them. Open acquires the native lifetime lock before copying the schema to Python for validation, so schema and physical metadata are one locked snapshot. Dense record width is derived once from dtype/item_shape and passed to the native handle as runtime geometry; it is not independently persisted as a second authority. There is no format version or legacy kind dispatch.

2. Physical header — the first 32 bytes of meta.zr. The format deliberately carries no payload or metadata checksum.

• codec is the store's one compression method, stamped at creation and immutable — there is no per-record tag anywhere.
• length (u64) is the physical record count; it equals logical length for both dense stores and inline ragged stores.
• tail_offset (u64) is the absolute committed frontier in data.zr.
• schema_length (u32) is the occupied prefix of the static schema area and must be in 1..4064.

const Codec = enum(u8) { raw = 0, zstd = 1, zstdict = 2, _ };
const Header = extern struct {
    length: u64,
    tail_offset: u64,
    schema_length: u32,
    reserved: [11]u8,
    codec: u8,
};

3. Record table — contiguous 16-byte entries start at offset 4096 in meta.zr and grow as location runs are written. offset is an absolute byte offset in data.zr; phys_length/logic_length are the stored and original sizes. The codec is not here: it is the header's, so a record is stored exactly the way the store is declared.

const RecordLoc = extern struct {
    offset: u64,
    phys_length: u32,
    logic_length: u32,
};

There is no liveness flag. Every entry below length is a record.

4. No fixed record-count cap. The table and payload stream grow naturally in their fixed files. The practical bounds are the u64 count, supported positional file offsets, 2 GiB per record, and disk.

Executor

1. Write. Writes are append-only; everything else is offset redirection. The codec is immutable store state, so append and gather dispatch once at their entry points into separate raw, zstd, or zstd-dictionary implementations. Their contexts and workers are deliberately not unified: only validation, location bounds, locking, and final publication are shared. Geometry is equally explicit across the whole stack: Python derives Dense record width from its schema and passes it to each fixed-stride operation, while Ragged supplies offsets for its shape-prefixed records. The private ABI turns those inputs into compile-time record sources and destinations; the engine has one append and one gather operation. Its shared opaque handle remains private and carries no typed-store geometry.

• Compressed append: workers claim individual records and reuse one operation-local frame buffer per task, reserve physical offsets in completion order through a short frontier lock, and issue positional payload writes in parallel. Logical IDs remain in the loc table, so physical completion order does not change random gather. The caller waits for every payload before publishing the new locations and in-process length. Append does not overwrite the on-disk Header and is intentionally lazy. Writer close() writes the current Header and closes the files, making the page-cache state available for read-only open. Every compressed frame owns one absolute range in data.zr; positional writes extend the file to the current frontier. Python budgets the process-wide executor at three quarters of the logical CPUs available to the process, leaving headroom for packing, transforms, and the caller without encoding a platform-specific thread count. Each producer configures its CCtx or shared immutable CDict once, then starts every independent record frame with ZSTD_compress2.
• Raw append preserves the stronger invariant already supplied by Python: every record in one Dense or packed Ragged batch is a boundary inside one contiguous source buffer. After locating the records, the engine consumes that buffer in plan order and passes each contiguous planned range directly to writePositionalAll on data.zr. There is no per-record iovec construction, byte-budget flush, payload copy, or platform-specific syscall path. Zig's std.Io handles short writes and maps the same positional operation to POSIX and Windows implementations. 2. Read. Fill the destination memory concurrently, in place from the Python side (executed on async threads).
• Committed records are immutable and length is published through an atomic. The fixed meta.zr and data.zr handles require no rollover fd-table synchronization; record I/O itself stays lock free.
• Every record is read at the offset its table entry records — the record table is addressed by pure arithmetic, so random access is one pread for the location and one for the bytes, with no batching assumptions about layout. Each lane reads one location and immediately reads/decompresses that record; there is no separate metadata phase or sequential-run special case. Compressed records use a per-lane staging buffer and decode in place into the destination.

3. Internal fan-out. Io.Group.async fans work out up to the executor lane budget; lanes for which the runtime cannot reserve concurrency run inline on the calling thread. Python configures the process-level budget as max(physical cores, logical cores * 3 / 4), using platform topology where available.

• Lanes receive contiguous blocks rather than a strided subset, keeping each worker's reads and writes sequential.
• Each lane creates one zstd context (ZSTD_CCtx to write, ZSTD_DCtx to read) and reuses it across every record it handles, rather than paying that setup per record. The dictionary (ZSTD_CDict/ZSTD_DDict) is immutable, so all lanes share one, lock-free.
• Decompression writes straight into the caller's destination buffer, so there is no intermediate copy.

4. File access.

• Metadata: one naturally growing meta.zr, containing the fixed Header/schema page and loc table.
• Payload: one naturally growing data.zr, accessed concurrently through readPositionalAll/writePositionalAll. No path depends on filesystem sparse-file support.

Execution model. Opened readers are immutable, so calls on the same reader may gather concurrently. Creator appends are synchronous and native Storage serialization protects their physical commit. close() requires a quiescent handle; it is not concurrent with append or gather. Each native append or gather fans out internally across the shared Executor. A creator holds a lifetime, nonblocking exclusive lock on meta.zr; opened readers hold shared locks, so multiple handles and processes may consume one completed container concurrently.