loaderx 0.4.1

A record-based data runtime, focused on delivering extreme throughput and low latency

Loaderx

A compact and high-performance single-machine data loader designed for JAX/Flax.

Only Linux_amd64

Design Philosophy

loaderx is built around several core principles:

1. A pragmatic approach that prioritizes minimal memory overhead and minimal dependencies.
2. A strong focus on single-machine training workflows.
3. We implement based on NumPy semantics, supporting Blosc2 backends.
4. An immortal (endless) step-based data loader, rather than the traditional epoch-based design—better aligned with modern ML training practices.

Zsampler

Index Generator: a high-performance sampler implemented in Zig

1. Sequential generation: indices are produced by traversing the index space in order.
   • Sliding traversal: indices are obtained using a fixed-size sliding window. Note that in this case, the index space is treated as a circular queue to avoid truncation at the tail.
2. Random generation: indices are sampled randomly from the index space.
   • Global random: a set of samples is drawn randomly from the entire index space.

Convert a NumPy tensor to Blosc2

import blosc2
import numpy as np

np_arr = np.load('arr.npy',mmap_mode='r')
b2_arr = blosc2.asarray(np_arr, urlpath="arr.b2nd", mode="w")

Then, you can use b2_arr or load as b2_arr = blosc2.open("arr.b2nd")

Current Limitations

Currently, loaderx only supports single-host environments and does not yet support multi-host training.

Quick Start

from loaderx import Dataset, DataLoader

dataset = Dataset('train_data.b2nd')
labelset = Dataset('train_label.b2nd')

loader = DataLoader(dataset, labelset)

for i, batch in enumerate(loader):
    if i >= 256:
        break

print(batch['data'].shape)
print(batch['label'].shape)

Integrating with JAX/Flax

For practical integration examples, please refer to the Data2Latent repository