dataio
dataio is a Rust-backed data loader and IO layer for Python. It reads and writes local files, S3, R2, GCS, Azure, HTTP(S) and the Hugging Face Hub, runs decode/transform/batch pipelines on native worker pools, and returns NumPy, PyTorch, JAX, DLPack or Arrow data. It supports free-threaded CPython and ships a command line that replaces s5cmd.
Install
pip install dataio-rs
python -m dataio # report this host's capabilities
Python 3.11 or newer on Linux x86_64 (glibc 2.28+) gets a prebuilt wheel: abi3 for CPython with the GIL, cp314t for free-threaded CPython 3.14; other platforms build from the sdist (Develop). Nothing else is required: install NumPy, PyTorch, JAX or PyArrow for the outputs you use. CUDA, nvJPEG, cuFile (GDS) and libfabric load on first use, so the wheels run without them. To upgrade, read the changelog and the migration guide.
Quick start
This runs without input files or a framework:
import dataio
records = [{"key": "a", "payload": b"red"}, {"key": "b", "payload": b"sky"}]
pipe = dataio.pipeline(dataio.slot("payload").decode_bytes(), dataio.batch(2))
with pipe.bind(records).run("dlpack") as loader:
for batch in loader:
print(batch.keys, batch["payload"].shape) # ['a', 'b'] [2, 3]
output= selects the framework. The loader's run() defaults to "torch" and
also takes "numpy", "jax", "dlpack", a name registered with
register_output_adapter, or a callable. Readers default to "numpy" (read_dcp
to "torch"): read_image takes the loader's choices, and tensor readers such as
read_safetensors take "numpy" or "torch". Column-batch iterators
(iter_batches, iter_arrow_batches) yield {column: [values]} unless
output="numpy". The examples are complete scripts.
Build a pipeline
A pipeline is optional record maps, one or more slot chains, one batch stage
(batch, collate or group_by), then optional batch transforms and maps.
Building it does no IO; .bind(records).run(...) starts a loader.
pipe = dataio.pipeline(
dataio.map(lambda row: {"image": row["path"], "label": row["class_id"]}),
dataio.slot("image").decode_image(mode="rgb").resize_short(size=256)
.center_crop(h=224, w=224).normalize("imagenet"),
dataio.slot("label").passthrough(),
dataio.batch(64, drop_last=True, prefetch=4),
)
with pipe.bind(records).run("torch", concurrency=64) as loader:
for batch in loader:
train_step(batch["image"], batch["label"])
Slot methods are shorthand for the factories in these namespaces:
| Namespace | Operations |
|---|---|
dataio.decoders |
Image, audio (resampling="sinc" anti-aliases), JSON, NPY, safetensors, text, bytes and PT decoding |
dataio.transforms |
Resize, crop, normalize, augment and conditional transforms |
dataio.batch_transforms |
MixUp, CutMix, padding, shuffle and normalization |
dataio.cond |
Field and shape predicates for slot activation and transforms |
A dataio.map before the batch stage gets input records, one after it the decoded
Batch: a passthrough slot plus dataio.map(lambda b: collate_fn(b["sample"]))
reuses an existing collator. collate() keeps the batches a source yields,
group_by(key) batches each run of equal keys, and
from_sampler(batch_sampler, row_fn) tags a batch sampler's rows for
group_by("batch_id"). Examples: batch boundaries,
conditional slots,
conditional transforms; contracts:
callbacks.
Choose sources
bind takes a reusable iterable or a zero-argument factory and reads it once per
epoch; from_iterable(...) is single-pass. Wrap a source to index, compose,
shuffle or mix it:
photos = dataio.records(dataset, indexed=True) # len() + integer __getitem__
art = dataio.webdataset(manifest="s3://b/art/manifest.json")
records = dataio.mix([photos, art], [0.7, 0.3], seed=0, stop="first")
| Call | Effect |
|---|---|
records(dataset, indexed=True) |
Integer row access, bypassing the dataset's iterator; with batch(N), __getitems__(indices) fetches rows together |
.map_records(fn), .flat_map_records(fn), .concat(*sources) |
Replace one record, emit zero or more, append sources without opening them early |
.shuffle(seed=42, buffer_size=4096) |
Mix nearby records in a buffer of that many records; no buffer_size means a full permutation |
mix(sources, weights, seed=, stop=) |
Interleave indexed sources by weight into an indexed source; stop="first" ends before a source repeats, "all" once every source was read, "never" after length= samples |
webdataset("shard-{0000..0099}.tar"), webdataset(manifest=...) |
Tar shards; a shard_writer manifest opens as an indexed, seekable source |
Expanded and streaming sources need a buffered shuffle. Contracts: record sources, bulk reads, weighted mix; examples: callable sources, composition.
Configure execution
Loader options and process-wide environment settings bound concurrency and
memory; dataio.env_vars() lists every setting with its default.
| Setting | Effect |
|---|---|
run(..., concurrency=N) |
Per-shard stage admission; start with the default "auto", measure higher values for high-latency sources |
batch(..., prefetch=K) |
Batches in flight |
run(..., max_prefetch_bytes=B) |
Logical bytes of pending native outputs with a fixed layout; not an RSS limit |
run(..., map_workers=N) |
Threads for record maps, sized-source __getitem__ and from_sampler record functions; output equals N=1, callbacks must be thread-safe |
run(..., order="completion") |
Deliver batches as they finish instead of in submit order; no mid-epoch resume |
DATAIO_CPU_WORKERS / DATAIO_IO_WORKERS |
Process-wide CPU / async IO worker pools |
DATAIO_HTTP_MAX_INFLIGHT_READS |
Process-wide GET/HEAD limit; 0 means unlimited |
DATAIO_OBJECT_SHARDS |
HTTP and object-store client shard fanout |
From async code, run async for batch in batches inside
async with loader.aiter() as batches: it reuses native prefetch and closes on
early exit. One iteration owns a loader at a time. Scheduling, memory bounds and
cleanup: plans and execution,
iteration and cancellation.
Handle errors
run(..., on_failure=...) decides what a failed row does; batch.errors lists each
failure (index, key, stage, op, message).
on_failure= |
Effect |
|---|---|
"skip" (default) |
Drop failed rows; raise when no row of a batch survives |
"strict" |
Raise on a failed row |
N |
Drop failed rows; raise when fewer than N rows of a batch survive |
log_errors=True logs each failure to the dataio.rows logger, or pass a callable.
Source-iteration and Python batch-map errors propagate as raised
(callback contracts).
Shard and resume
shard= splits each epoch across ranks; state_dict() / load_state_dict()
checkpoint a loader mid-epoch. A keyed shuffle reshuffles a huge indexed dataset
every epoch without materializing its index, and restores on any world size:
records = dataio.records(dataset, indexed=True, resume="seek")
loader = pipe.bind(records).run(
"torch", shard={"id": rank, "of": world_size}, seed=42, shuffle="keyed",
) # or shuffle=("block", 1024, 8) to read 1024-record blocks within 8-block windows
states = [None] * world_size
torch.distributed.all_gather_object(states, loader.state_dict()) # save `states`
# After restarting on any world size, every rank restores the gathered list:
loader.load_state_dict(states)
Ranks save after the same step; on an unchanged layout a rank can restore its own
state_dict() alone. "pad": True gives every rank the same batch count. A
restore replays the consumed prefix unless resume="seek" lets an indexed source
with index-determined lookups skip it. Contracts:
resume, checkpoint identity,
elastic resume.
Read and write objects
Local paths, s3://, r2://, gs://, az://, hf:// and HTTP(S) URIs share
these operations where the backend supports them:
dataio.configure_endpoint("r2", "https://<account>.r2.cloudflarestorage.com")
dataio.configure_transport("https", timeout_secs=30, readall_concurrency=32)
chunk = dataio.read("s3://bucket/file.bin", offset=0, size=1 << 20)
parts = dataio.fetch("s3://bucket/file.bin", ranges=[(0, 64), (4096, 256)]) # one GET
dataio.write("s3://bucket/out.bin", chunk, if_none_match="*") # create only
etag = dataio.copy("s3://bucket/out.bin", "s3://bucket/copy.bin")
dataio.exists("s3://bucket/out.bin") # False only when missing; other errors raise
dataio.ls("s3://bucket/prefix/", recursive=False, detail=True) # one level, size/etag/mtime
dataio.glob("s3://bucket/**/*.safetensors")
payload = dataio.reader("s3://bucket/shard.tar.gz").gunzip().tar_entry("0001.jpg").read_all()
with dataio.writer("s3://bucket/checkpoint.bin") as writer:
writer.write_chunks(huge_generator())
with dataio.shard_writer("s3://bucket/out/shard-{:06d}.tar", max_bytes=1 << 30) as shards:
for key, jpg, meta in samples():
shards.write({"__key__": key, "jpg": jpg, "json": meta})
records = dataio.webdataset(manifest=shards.manifest) # indexed; no tar headers read
fetch_into(uri, buffer) fills an existing buffer, stream(uri) reads a body on
demand (bytes, lines or server-sent events), head_many(uris) sends HEADs
concurrently, and copy(recursive=True), sync and delete(recursive=True) work
on trees like the command line. Requests retry throttling, 5xx,
resets and timeouts; max_attempts= and deadline_secs= bound one call.
On s3:// and r2://, the store verifies every upload's CRC64NVME, and every
whole-object download is checked against the stored one.
s3:// uses the AWS credential chain (environment, profiles, SSO, web identity,
ECS/EKS Pod Identity, instance metadata); configure_credentials(scheme, ...) sets
explicit credentials, and clear_credentials(scheme) re-reads them. Details:
network configuration,
byte and format IO.
Command line
python -m dataio runs the same operations with the same credentials, so shell
scripts need no other S3 tool; python -m dataio <command> --help lists its flags.
python -m dataio cp model.safetensors s3://bucket/models/ # -> .../models/model.safetensors
python -m dataio cp -r --workers 32 s3://bucket/models/vae /nvme/vae
python -m dataio cp -n 's3://bucket/ckpt/__3_*.distcp' /nvme/ckpt/
python -m dataio sync --workers 128 --delete --no-follow-symlinks /fsx/run s3://bucket/archive/run
python -m dataio rm -r s3://bucket/scratch/run-1 # prints "deleted 1234"
python -m dataio sync --dry-run --delete --exclude '*.tmp' /fsx/run s3://bucket/run # plan only
python -m dataio --json sync s3://bucket/data /nvme/data # {"copied": .., "skipped": .., "deleted": ..}
python -m dataio head --max-attempts 1 --deadline 5 s3://bucket/ckpt/COMMIT # a fast probe
| Command or flag | Effect |
|---|---|
cp SRC DST |
Copy a file or object (into DST/<name> when DST ends in / or is a local directory), a directory or prefix with -r, or every match of a wildcard |
sync SRC DST |
Copy the files of a directory or prefix that are missing or differ under DST |
rm URI |
Delete a file or object, or a directory or prefix with -r |
head URI, cat URI, ls URI |
Print metadata (exit 1 when missing), the body (-o FILE saves it), or the URIs below a prefix (--long adds sizes) |
info (the default), parquet PATH, safetensors PATH |
Print host capabilities, Parquet metadata or safetensors entries |
cp -n |
Skip destinations that exist |
cp --etag ETAG |
Copy one object only if it still has this ETag |
sync --delete |
Remove destination files the source lacks |
sync --strategy S |
size (default) compares sizes; mtime also copies files modified after their copy, between local trees; checksum also compares equal-size files by CRC64NVME (s3, r2, local) |
sync --no-follow-symlinks |
Skip symlinks in a local source |
--include PATTERN, --exclude PATTERN (cp -r or a wildcard, sync, rm -r) |
Select files by their path relative to the source: an include pattern, if any, must match and no exclude pattern may (* also matches /; repeatable) |
--dry-run (cp, sync, rm) |
Print each planned copy SRC DST / delete URI and the summary; change nothing |
cp - DST |
Stream stdin to DST (-n: create only) |
--workers N (cp, sync) |
Transfers in flight (default 64) |
--max-attempts N (not info, parquet, safetensors) |
Attempts per request on throttling, 5xx, resets and timeouts (default DATAIO_OBJECT_MAX_ATTEMPTS, 4) |
--deadline SECS (likewise) |
One bound for the whole command: listing, transfers, retries and backoff |
--json (before the command) |
Print JSON instead of text |
Output. cp, sync and rm -r print one summary line (copied 3, skipped 1),
or one JSON object with --json, and nothing per file. A failure prints one stderr
line and exits 1; the first failure stops the remaining transfers, and re-running
sync resumes. A usage error exits 2. Printed URLs drop userinfo and query
strings, so a presigned signature never reaches a terminal or log.
Recursive commands treat a prefix as a directory (s3://b/run never reaches
s3://b/run2/), and sync refuses a source with no file, so a mistyped source
never empties the destination under --delete. Wildcards, filters, dry runs, path
checks and retry limits: the command-line contract.
Replacing s5cmd 2.2.2:
| s5cmd | dataio |
|---|---|
cp 's3://b/p/*' dir/, cp 'dir/*' s3://b/p/ |
cp -r s3://b/p dir, cp -r dir s3://b/p (local hidden files included) |
cp 's3://b/p/*.parquet' dir/, cp -n 's3://b/p/__3_*' dir/ |
the same pattern (-n as well) |
cp FILE s3://b/p/, cp s3://b/k /tmp/f |
the same |
sync 's3://b/p/*' dir/, sync dir/ s3://b/p/ |
sync s3://b/p dir, sync dir s3://b/p (no wildcards) |
--numworkers N |
--workers N (default 64) |
--retry-count N |
--max-attempts N+1 (per command; default 4 attempts) |
timeout SECS s5cmd ... |
--deadline SECS (exit 1 with a TimeoutError line) |
sync --size-only |
default --strategy size |
sync --size-only, then an ls count/bytes check |
sync --strategy checksum: exit 0 means every file's content matches |
sync (size + newer mtime), local trees |
--strategy mtime |
sync --delete, --no-follow-symlinks |
the same flags |
--exclude P, --include P (cp, sync, rm) |
the same flags (paths relative to the source; * also matches /) |
--dry-run |
the same flag (copy / delete lines, then the summary line) |
--exit-on-error |
always |
--log error, --stat |
default: no per-file lines, one summary line |
--concurrency N, --part-size MiB (per-file parts) |
DATAIO_MULTIPART_CONCURRENCY (upload parts in flight per process, 64) / DATAIO_READALL_CONCURRENCY (ranges per download, 32), DATAIO_MULTIPART_PART_MIB (16; parts grow past it to fit 10 000) / DATAIO_READALL_CHUNK_MAX_MIB |
rm 's3://b/p/*', rm 's3://b/p/*.tmp' |
rm -r s3://b/p, rm -r --include '*.tmp' s3://b/p |
ls 's3://b/p/*' (count, bytes) |
--json ls --long s3://b/p/ (count, bytes) |
cat s3://b/k |
cat s3://b/k |
pipe DST |
cp - DST (-n: create only) |
ls s3://b/k as an existence test |
head s3://b/k (exit 1 when missing), or dataio.exists in Python |
run FILE (a batch of commands) |
not a command: call dataio.copy from Python, or cp -r / a wildcard |
credential_process shim for Pod Identity |
not needed |
Read structured formats
Typed readers return arrays, images, columns and Arrow data without a pipeline:
array = dataio.read_numpy("/data/x.npy")
image = dataio.read_image("s3://bucket/image.png", mode="rgb")
columns = dataio.read_parquet("/data/shard.parquet", columns=["key", "height"])
tall = dataio.scan_parquet("/data/shard.parquet").filter(lambda r: r["height"] > 512).collect()
stream = dataio.scan_feather("/data/table.arrow").select(["key"]).collect_arrow_stream()
for batch in dataio.scan_parquet("/data/features.parquet").iter_batches(output="numpy"):
features, tokens = batch["features"], batch["tokens"] # ndarray, RaggedArray
for example in dataio.iter_tfrecord("/data/train.tfrecord"):
label = example["label"]
Scans keep Arrow types (collect_arrow_stream(), or PyArrow through
collect_arrow_reader() / collect_arrow()). iter_batches(output="numpy") needs
only NumPy: structs become dicts, fixed-size lists add dimensions, and
variable-length lists become RaggedArray(values, offsets) (row i is
values[offsets[i]:offsets[i + 1]]). iter_arrow_batches(source, output="numpy")
does the same for any __arrow_c_stream__ producer, such as a DuckDB relation.
Types, nulls and views: columnar NumPy batches.
Read or fill safetensors weights
Single-file and sharded checkpoints use the same API:
dataio.write_safetensors("checkpoints/model/", state_dict, max_shard_size="5GB")
subset = dataio.read_safetensors("checkpoints/model/", keys=["layer.weight"], output="torch")
with dataio.open_safetensors("weights.safetensors") as source:
tensor = source.read(keys="layer.weight", output="torch")["layer.weight"]
source.read_into(destinations, chunk_bytes=16 << 20)
read_into checks every shape, dtype and layout, then fills existing NumPy arrays
or CPU/CUDA Torch tensors (destinations maps names to them) in bounded chunks,
without another full copy. On local GDS-ready storage,
read_safetensors(..., output="torch", device="cuda:0") reads straight to the GPU,
falling back to CPU with a warning; gds_available() and gds_info() report
support. Views, copies and CUDA staging: buffer ownership.
Convert distributed checkpoints
scan_dcp reassembles a PyTorch distributed checkpoint (DCP) into safetensors;
read_dcp returns its tensors:
dataio.scan_dcp("s3://bucket/checkpoint", keys="model.", bf16=True).rename(
lambda name: name.removeprefix("model.")
).write("weights.safetensors", strategy="stream")
DCP metadata is a Python pickle: load only trusted checkpoints. Selection, streaming and layouts: distributed checkpoints.
Transfer with RDMA
dataio.rdma does one-sided reads and writes through libfabric, loaded on first use:
import dataio.rdma as rdma
with rdma.serve(bind="0.0.0.0:0", buffer_size=1 << 20) as server:
with rdma.connect(server.local_addr) as client:
client.write_bytes(0, b"hello rdma")
assert client.read_bytes(0, 10) == b"hello rdma"
serve(bind=..., file=path) exposes file ranges; rdma.read_into fills
caller-owned CPU or CUDA buffers. available() also succeeds over TCP software
providers, so check info() or diagnose() for physical RDMA. GPUDirect RDMA
needs an HMEM provider and libfabric API 1.9+ (RDMA transport).
API index
| Task | Entry points |
|---|---|
| Build a pipeline | pipeline, slot, slots, chain, batch, collate, group_by |
| Extend processing | map, custom_op, register_output_adapter |
| Bind records | records, files, urls, manifest, from_callable, from_iterable, from_sampler, webdataset, archive, mix |
| Work with bytes | read, fetch, fetch_into, write, copy, sync, head, head_many, stream, ls, glob, exists, delete |
| Compose IO | reader, writer, shard_writer, scan, from_iterator |
| Work with tensors | read_safetensors, open_safetensors, write_safetensors, scan_dcp, read_dcp, scan_pt, read_pt |
| Work with arrays/tables | read_* / scan_* / write_* for numpy, npz, parquet, feather, tfrecord; iter_arrow_batches, read_image, iter_tfrecord |
| Configure IO | configure_credentials, configure_endpoint, configure_transport, clear_credentials |
| Use GDS | gds_open, gds_register_buffer, gds_read_into_tensor |
The documentation index links the contracts, the benchmark guide and the roadmap.
Develop
uv run --no-sync --with maturin maturin develop --profile staging
cargo test
uv run --no-sync python -m pytest python/tests -q
uv run --no-sync --with maturin maturin build --profile staging --out dist # a wheel
The toolchain is pinned in rust-toolchain.toml. Source builds omit RDMA: install
libfabric headers and add --features rdma (published wheels include it). Under a
free-threaded interpreter (uv venv --python 3.14t) the same commands build the
cp314t extension; maturin build -i python3.11 python3.14t builds both wheels. The
native IO, network, Arrow and format crates build without Python.
License
Apache-2.0. Bundled dependency licenses are in THIRD_PARTY_NOTICES.
Metadata
Release files for dataio-rs 0.5.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| dataio_rs-0.5.4.tar.gz | 1.4 MB | Without provenance | Details for dataio_rs-0.5.4.tar.gz |
Built distributions (wheels)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| dataio_rs-0.5.4-cp314-cp314t-manylinux_2_28_x86_64.whl | CPython 3.14 | CPython 3.14 free-threading | Linux glibc 2.28+ x86-64 | Details for dataio_rs-0.5.4-cp314-cp314t-manylinux_2_28_x86_64.whl |
| dataio_rs-0.5.4-cp311-abi3-manylinux_2_28_x86_64.whl | CPython 3.11 | abi3 | Linux glibc 2.28+ x86-64 | Details for dataio_rs-0.5.4-cp311-abi3-manylinux_2_28_x86_64.whl |
Total release size: 30.3 MB
Release files / dataio_rs-0.5.4.tar.gz
| Download URL | dataio_rs-0.5.4.tar.gz |
|---|---|
| Size | 1.4 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8a9c3d0719b482e0fce9e42c1fbd521a03106462fb5087228a673bbc4c487453
|
|
BLAKE2b-256 checksum How to use checksums |
98e6707d7fd9b5985f217004e312bfb9974edc55053cacafb242ba7ce8dc473b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / dataio_rs-0.5.4-cp314-cp314t-manylinux_2_28_x86_64.whl
| Download URL | dataio_rs-0.5.4-cp314-cp314t-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 14.4 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
5bb654e2cbd519ccad1081eb7f2e378d78f51dab203bcc0f2aa47b052d7201b5
|
|
BLAKE2b-256 checksum How to use checksums |
6514f3a1fe294d291a7640a43165e4cdcff4acef182ce204fb3793490fb372a8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / dataio_rs-0.5.4-cp311-abi3-manylinux_2_28_x86_64.whl
| Download URL | dataio_rs-0.5.4-cp311-abi3-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 14.4 MB |
| Tags | CPython 3.11 Linux glibc 2.28+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
70bcee77b263e8259ec5ba715631870deebf63a6d120a95dbdc3abf4d4edfb4d
|
|
BLAKE2b-256 checksum How to use checksums |
5f7d74a4fd37f871bb918ac862e689320c707ce4f6c6e7f20122c3f16b2368da
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Security for dataio-rs 0.5.4
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.