dataio
A Rust-backed data-loading library for local files, object stores, tensors and columnar data. Build reusable fetch/decode/transform/batch pipelines or read individual objects through the same IO layer. Outputs include NumPy, PyTorch, DLPack and Arrow.
Native IO and kernels release the GIL and share worker pools. Python source and map callbacks run at explicit boundaries. Buffer sharing depends on ownership, layout and device transfer; retained views keep their backing storage alive.
Install and run
pip install dataio-rs
python -m dataio # report this host's capabilities
Python 3.11 or newer is required. There are no required Python dependencies; install NumPy, PyTorch or PyArrow when using that framework's outputs. Wheels run without CUDA libraries. CUDA, nvJPEG, cuFile/GDS and libfabric are loaded when their features are used.
0.5.0 is a breaking release: see the changelog and the migration guide when upgrading from 0.4.x.
This example runs without input files or an output framework:
import dataio
records = [
{"key": "a", "payload": b"red"},
{"key": "b", "payload": b"sky"},
]
pipe = dataio.pipeline(
dataio.slot("payload").decode_bytes(),
dataio.batch(2),
)
with pipe.bind(records).run("dlpack") as loader:
for batch in loader:
print(batch.keys, batch["payload"].shape) # ['a', 'b'] [2, 3]
Use .run("numpy") or .run("torch") to convert tensor slots to that framework.
The following examples show common tasks; file paths, datasets and callbacks
must be supplied by your application. For complete scripts, see Examples.
Build a pipeline
A pipeline contains optional record maps, one or more slot chains, exactly one
batching stage (batch, collate or group_by), then optional batch transforms
and Python maps. Creating the plan does no IO; .bind(records).run(...) creates
the loader. Close loaders with a context manager, especially when stopping early.
pipe = dataio.pipeline(
dataio.map(lambda row: {"image": row["path"], "label": row["class_id"]}),
dataio.slot("image")
.decode_image(mode="rgb")
.resize_short(size=256)
.center_crop(h=224, w=224)
.normalize("imagenet"),
dataio.slot("label").passthrough(),
dataio.batch(64, drop_last=True, prefetch=4),
)
with pipe.bind(records).run("torch", concurrency=64) as loader:
for batch in loader:
train_step(batch["image"], batch["label"])
Slot methods are shorthand for the factories in these namespaces:
| Namespace | Operations |
|---|---|
dataio.decoders |
Image, audio, JSON, NPY, safetensors, text, bytes and PT decoding |
dataio.transforms |
Resize, crop, normalize, augment and conditional transforms |
dataio.batch_transforms |
MixUp, CutMix, padding, shuffle and normalization |
dataio.cond |
Field and shape predicates for slot activation and transforms |
For example, .decode_audio(sample_rate=16000, resampling="sinc") enables
anti-aliased resampling; the default "linear" costs less computation.
Conditional slots can gate optional inputs
before decode; conditional transforms branch
on decoded shapes.
Choose and compose sources
Bind a reusable iterable or a zero-argument record factory for repeated passes.
from_iterable(...) is single-pass, including when given a factory. For a finite
third-party dataset with integer row access, choose
dataio.records(dataset, indexed=True). It requires __len__ and integer
__getitem__ and bypasses the dataset's iterator.
With batch(N), indexed datasets can provide __getitems__(indices) to fetch
rows together. Return a list or tuple of exactly the requested rows, in order,
including duplicate indices. Fetch errors propagate without scalar retries.
See the bulk-read protocol and
runnable example.
Source transforms run before native decoding:
def expand(record):
for index, payload in enumerate(record["parts"]):
yield {"key": f"{record['key']}:{index}", "payload": payload}
records = dataio.records(group_a).concat(group_b).flat_map_records(expand)
map_records(fn) replaces one record; flat_map_records(fn) emits zero or more
complete records; concat(...) appends sources without opening later ones early.
Expansion has unknown length. Use bounded shuffle for expanded or streaming
sources:
records = dataio.webdataset("/data/shard-{0000..0099}.tar").shuffle(
seed=42, buffer_size=4096,
)
The buffer bounds pending records, not bytes, and mixes nearby records. Omitting
buffer_size requests a full permutation, materializing non-indexed sources;
single-pass sources require a buffer. A fixed seed repeats the order only for an
unchanged, repeatable source. See source contracts.
Mix indexed sources by weight. The mix is itself indexed, so sharding, loader
shuffling and resume="seek" apply to it:
records = dataio.mix(
[dataio.records(photos, indexed=True), dataio.webdataset(manifest="s3://b/art/manifest.json")],
[0.7, 0.3], seed=0, stop="first",
)
Sources interleave evenly in proportion to the weights. stop="first" ends the
epoch before any source repeats a row, "all" once every source was read
(wrapping the others) and "never" after length= samples; each epoch continues
every source where the last one stopped. See the mix contract.
Adapt Python collation and sampler batches
Python maps before batching see input records. Maps after batching see the
decoded Batch and may return nested mappings, tuples or application objects.
They run on the consumer thread during synchronous iteration. Native batch
transforms must precede Python batch maps.
Pass Python objects through a slot to reuse an existing collator:
pipe = dataio.pipeline(
dataio.slot("sample").passthrough(),
dataio.batch(32),
dataio.map(lambda batch: collate_fn(batch["sample"])),
)
Each input record must contain sample. For native tensor slots, a batch map
can instead assemble nested results from batch["slot_name"].
collate() preserves batch boundaries supplied by the source; it does not take
a Python collate function. group_by(key) emits one batch for each contiguous
run of a key. To adapt a sampler yielding lists of rows:
records = dataio.from_sampler(
batch_sampler,
lambda row: {"key": row["id"], "payload": row["path"], "meta": row},
)
pipe = dataio.pipeline(
dataio.slot("payload").decode_npy(),
dataio.slot("meta").passthrough(),
dataio.group_by("batch_id", prefetch=4),
)
from_sampler adds batch_id to shallow copies of record mappings. Omit its
second argument when the sampler already yields valid records. The
batch-boundary example compares both approaches.
Consume asynchronously
with pipe.bind(records).run("numpy") as loader:
async with loader.aiter() as batches:
async for batch in batches:
await consume(batch)
The async iterator reuses native prefetch and closes on early context exit. Source and batch callbacks run on its worker thread. Cancellation waits for an already-running synchronous callback to return. Do not mix sync and async iteration on the same loader. Checkpoints track batches delivered to the caller; see iteration and cancellation.
Configure execution
| Setting | Scope |
|---|---|
run(..., concurrency=N) |
Per-shard stage admission; total activity also depends on prefetch |
batch(..., prefetch=K) |
Number of batches in flight |
run(..., max_prefetch_bytes=B) |
Logical bytes of pending native preallocated outputs |
DATAIO_CPU_WORKERS / DATAIO_IO_WORKERS |
Process-wide CPU / async IO worker pools |
DATAIO_HTTP_MAX_INFLIGHT_READS |
Process-wide GET/HEAD limit; 0 means unlimited |
DATAIO_OBJECT_SHARDS |
HTTP and object-store client shard fanout |
Start with concurrency="auto"; measure higher values for high-latency sources.
Submit order is the default. order="completion" returns completed batches sooner
but cannot resume mid-epoch. The output byte limit requires a fixed native layout
and direct writer; unsupported outputs and oversized batches fail before allocation.
It excludes scratch space, caches and consumer-held outputs, so it is not an RSS
limit. Architecture describes the scheduling
and ownership rules.
Handle errors
run(..., on_failure="strict") raises on a row error. "skip" drops failed rows
and raises when no rows survive; an integer requires that many surviving rows.
batch.errors retains per-row errors. Pass log_errors=True to use the
dataio.rows logger, or provide a logging callable.
Source-iteration and Python batch-map errors propagate directly. A failed batch map does not roll back consumed progress or its side effects. Padding and batch normalization preserve row metadata; MixUp and CutMix invalidate it because they combine rows. See callback contracts.
Shard and resume
loader = pipe.bind(records).run(
"torch", shard={"id": rank, "of": world_size, "pad": True}, seed=42,
)
# Save loader.state_dict() and restore with loader.load_state_dict(state).
pad=True equalizes rank batch counts. Resume replays a partial batch to align
native transform RNG. By default it re-enumerates the consumed source prefix;
source data and callbacks must replay consistently. A checkpoint identifies the
plan and tracked source configuration, not hidden callback state or data contents.
Deterministic indexed datasets can avoid fetching consumed rows:
records = dataio.records(dataset, indexed=True, resume="seek")
Seek requires lookups determined by index. Source maps, source state hooks and composed iterables retain replay. See the resume contract and checkpoint compatibility.
Shuffle a huge indexed dataset every epoch without materializing its index, and resume on a different number of ranks (resuming elsewhere skips the consumed prefix, so the source must be able to seek):
loader = pipe.bind(dataio.records(dataset, indexed=True, resume="seek")).run(
"torch", shard={"id": rank, "of": world_size}, seed=42, shuffle="keyed",
) # or shuffle=("block", 1024, 8) to read 1024-record blocks within 8-block windows
states = [None] * world_size
torch.distributed.all_gather_object(states, loader.state_dict()) # save `states`
# After restarting on any world size, every rank restores the gathered list:
loader.load_state_dict(states)
The ranks must save after the same step. The restored epoch continues where the old ranks stopped, rounded down to whole batches; see keyed shuffles and elastic resume.
Read and write objects
Local paths, s3://, r2://, gs://, az://, hf:// and HTTP(S) URIs share
these operations where the backend supports them:
chunk = dataio.read("s3://bucket/file.bin", offset=0, size=1 << 20)
dataio.write("s3://bucket/out.bin", chunk)
dataio.copy("s3://bucket/out.bin", "/data/out.bin")
dataio.head("s3://bucket/out.bin")
dataio.exists("s3://bucket/out.bin")
dataio.ls("s3://bucket/prefix/")
dataio.glob("s3://bucket/**/*.safetensors")
exists returns False only for a missing resource. Permission failures,
timeouts and other IO errors propagate, so an inaccessible object is not
mistaken for one that needs creating.
Fill an existing writable buffer with dataio.fetch_into(uri, buffer). For
composable reads and incremental writes:
src = dataio.reader("s3://bucket/shard.tar.gz")
payload = src.gunzip().tar_entry("0001.jpg").read_all()
with dataio.writer("s3://bucket/checkpoint.bin") as writer:
writer.write_chunks(huge_generator())
Write WebDataset shards and a manifest that later opens them as an indexed source without reading tar headers:
with dataio.shard_writer("s3://bucket/out/shard-{:06d}.tar", max_bytes=1 << 30) as shards:
for key, jpg, meta in samples():
shards.write({"__key__": key, "jpg": jpg, "json": meta})
records = dataio.webdataset(manifest=shards.manifest) # len(), indexed, seekable
Each shard commits atomically (local rename or completed multipart upload) and the manifest commits last. See shards and manifests.
reader.read_buffer() returns a read-only buffer retaining native storage.
dataio.stream(uri) reads the response body on demand; byte, line and SSE
iterators share a cursor. Close unused streams. Streaming does not retry a body
after exposing it to the caller; only read_all() materializes all remaining bytes.
Configure credentials and transport before use:
dataio.configure_credentials("s3", method="default")
dataio.configure_endpoint("r2", "https://<account>.r2.cloudflarestorage.com")
dataio.configure_transport(
"https", timeout_secs=30, readall_chunk_mib=32, readall_concurrency=8,
)
method="default" uses the platform credential chain. Repeating an identical
configure_credentials or configure_endpoint call is a no-op that keeps cached
clients and in-flight reads; to refresh environment- or file-based credentials,
call dataio.clear_credentials("s3") and configure again. Range settings apply to
each read; splitting requires a strong validator. Transport configuration
replaces the scheme's explicit options, so pass options together. Proxy, TLS,
cache, timeout and shared-read-limit behavior is documented in
network configuration.
Read structured formats
array = dataio.read_numpy("/data/x.npy")
image = dataio.read_image("s3://bucket/image.png", mode="rgb")
columns = dataio.read_parquet("/data/shard.parquet", columns=["key", "height"])
rows = dataio.scan_parquet("/data/shard.parquet").filter(
lambda row: row["height"] > 512,
).collect()
for example in dataio.iter_tfrecord("/data/train.tfrecord"):
label = example["label"]
Parquet and Feather/Arrow IPC scans preserve Arrow types without creating Python objects for individual cells:
scan = dataio.scan_feather("/data/table.arrow").select(["key", "value"])
stream = scan.collect_arrow_stream() # Arrow C stream protocol
reader = scan.collect_arrow_reader() # optional PyArrow batch reader
table = scan.collect_arrow() # optional PyArrow table
For numeric columns, iter_batches(output="numpy") yields arrays directly and
needs only NumPy:
scan = dataio.scan_parquet("/data/features.parquet").select(["features", "tokens"])
for batch in scan.iter_batches(output="numpy"):
features = batch["features"] # owned, writable, C-contiguous NumPy array
tokens = batch["tokens"] # dataio.RaggedArray(values, offsets)
Struct columns become nested dicts of arrays and fixed-size lists add array
dimensions. Variable-length numeric lists become RaggedArray(values, offsets):
row i is values[offsets[i]:offsets[i + 1]], with zero-based int64 offsets
and no padding. Selected null values and unsupported types raise errors.
copy=False returns read-only views that retain their buffers. See the
array ownership contract for supported
types and retention limits.
dataio.iter_arrow_batches(source, output=...) yields the same batches from any
CPU Arrow table stream exposing __arrow_c_stream__. For example, with optional
DuckDB installed:
import duckdb
query = duckdb.sql("SELECT range AS id FROM range(4)")
for batch in dataio.iter_arrow_batches(query, output="numpy"):
print(batch["id"]) # [0 1 2 3]
Pass a zero-argument factory to create the stream when iteration starts, such as
dataio.iter_arrow_batches(make_query); an existing reader can usually be
consumed only once. Arrow and NumPy outputs support projection and row slicing,
and Parquet also native expressions; Python callable filters need
output="python" or row collection. NPY decoding supports complex values,
foreign byte order and Fortran layout.
Read or fill safetensors weights
Single-file and sharded checkpoints use the same API:
dataio.write_safetensors(
"checkpoints/model/", state_dict, max_shard_size="5GB", metadata={"format": "pt"},
)
subset = dataio.read_safetensors(
"checkpoints/model/", keys=["layer.weight", "layer.bias"], output="torch",
)
with dataio.open_safetensors("weights.safetensors") as source:
tensor = source.read(keys="layer.weight", output="torch")["layer.weight"]
source.read_into(destinations, chunk_bytes=16 << 20)
destinations maps tensor names to existing NumPy arrays or Torch tensors.
read_into validates shape, dtype, layout and overlap before writing, then fills
CPU or CUDA targets in bounded chunks. It avoids another complete weight copy
and does not use GDS. IO failure can leave targets partially written; inference
Torch targets require an inference-mode context.
NumPy views are read-only; local Torch mappings use copy-on-write. The
ownership reference explains copies,
CUDA staging and buffer lifetime. Complex64 outputs are supported. On local
GDS-ready storage, pass output="torch", device="cuda:0" to
read_safetensors for direct GPU reads; unavailable GDS falls back to CPU with
a warning. gds_available() and gds_info() report the host's GDS support.
Convert distributed checkpoints
dataio.scan_dcp("s3://bucket/checkpoint", keys="model.", bf16=True).rename(
lambda name: name.removeprefix("model.")
).write("weights.safetensors", strategy="stream")
Selection uses original names; rename collisions fail before payload reads. Streaming batches whole tensors, so one tensor can exceed the batch target. Dense row-block shards and transposed storage are supported; storage transforms and other sharding layouts are rejected. DCP metadata uses Python pickle: load only trusted checkpoints. NumPy collection rejects BF16 instead of changing dtype.
Transfer with RDMA
dataio.rdma provides one-sided reads and writes through libfabric. It loads the
library on first use; available() can also succeed with TCP software providers,
so inspect info() or diagnose() to establish physical RDMA capability.
import dataio.rdma as rdma
with rdma.serve(bind="0.0.0.0:0", buffer_size=1 << 20) as server:
with rdma.connect(server.local_addr) as client:
client.write_bytes(0, b"hello rdma")
assert client.read_bytes(0, 10) == b"hello rdma"
serve(bind=..., file=path) exposes file ranges; rdma.read_into fills
caller-owned CPU or CUDA buffers. GPUDirect RDMA requires a compatible HMEM
provider and libfabric API 1.9 or newer. See RDMA transport for buffer,
provider and diagnostic details.
API index
| Task | Entry points |
|---|---|
| Build a pipeline | pipeline, slot, slots, chain, batch, collate, group_by |
| Extend processing | map, custom_op, register_output_adapter |
| Bind records | records, files, urls, manifest, from_callable, from_iterable, from_sampler, webdataset, archive, mix |
| Work with bytes | read, fetch_into, write, copy, sync, head, stream, ls, glob, exists, delete |
| Compose IO | reader, writer, shard_writer, scan, from_iterator |
| Work with tensors | read_safetensors, open_safetensors, write_safetensors, scan_dcp, read_dcp, scan_pt, read_pt |
| Work with arrays/tables | read_* / scan_* for numpy, npz, parquet, feather, tfrecord; iter_arrow_batches, read_image, iter_tfrecord |
| Configure IO | configure_credentials, configure_endpoint, configure_transport |
| Use GDS | gds_open, gds_register_buffer, gds_read_into_tensor |
The documentation index links to execution contracts, benchmarks and planned capabilities. Use the benchmark guide to compare the output and workload your application actually uses.
Develop
The toolchain is pinned in rust-toolchain.toml. Build the editable extension,
then run the relevant tests:
uv run --no-sync --with maturin maturin develop --profile staging
cargo test
uv run --no-sync python -m pytest python/tests -q
Source builds omit RDMA by default. Install libfabric development headers and add
--features rdma to enable it; published wheels include the feature. Native IO,
network, Arrow and format crates can build without Python. Build a wheel with:
uv run --no-sync --with maturin maturin build --profile staging --out dist
See Benchmarking for workload validation and comparisons.
License
Apache-2.0. Bundled dependency licenses are in THIRD_PARTY_NOTICES.
Metadata
Release files for dataio-rs 0.5.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| dataio_rs-0.5.0.tar.gz | 1.3 MB | Without provenance | Details for dataio_rs-0.5.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| dataio_rs-0.5.0-cp311-abi3-manylinux_2_28_x86_64.whl | CPython 3.11 | abi3 | Linux glibc 2.28+ x86-64 | Details for dataio_rs-0.5.0-cp311-abi3-manylinux_2_28_x86_64.whl |
Total release size: 14.3 MB
Release files / dataio_rs-0.5.0.tar.gz
| Download URL | dataio_rs-0.5.0.tar.gz |
|---|---|
| Size | 1.3 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3c2276ec98d659ca3091dfe3f59a8d7586da8473c3240fca633c9aac863d308a
|
|
BLAKE2b-256 checksum How to use checksums |
64b59a01873eed41e4c75b4f1810a6746179ac543db0c5395201da1944344e45
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / dataio_rs-0.5.0-cp311-abi3-manylinux_2_28_x86_64.whl
| Download URL | dataio_rs-0.5.0-cp311-abi3-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 13.0 MB |
| Tags | CPython 3.11 Linux glibc 2.28+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
058d0d2af366acd629b629d03c0ca52e958ebecca89a2ff6709ab345258b660c
|
|
BLAKE2b-256 checksum How to use checksums |
d4f33c968b82b6c9597d615d2eee5af2f72f41018e325cff6abe8b7bc9d66d6f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Security for dataio-rs 0.5.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.