Skip to main content

dataio

CI PyPI Python

dataio is a Rust-backed data loader and IO layer for Python. It reads and writes local files, S3, R2, GCS, Azure, HTTP(S) and the Hugging Face Hub, runs decode/transform/batch pipelines on native worker pools, and returns NumPy, PyTorch, JAX, DLPack or Arrow data. It supports free-threaded CPython and ships a command line that replaces s5cmd.

Install

pip install dataio-rs
python -m dataio  # report this host's capabilities

Python 3.11 or newer on Linux x86_64 (glibc 2.28+) gets a prebuilt wheel: abi3 for CPython with the GIL, cp314t for free-threaded CPython 3.14; other platforms build from the sdist (Develop). Nothing else is required: install NumPy, PyTorch, JAX or PyArrow for the outputs you use. CUDA, nvJPEG, cuFile (GDS) and libfabric load on first use, so the wheels run without them. To upgrade, read the changelog and the migration guide.

Quick start

This runs without input files or a framework:

import dataio

records = [{"key": "a", "payload": b"red"}, {"key": "b", "payload": b"sky"}]
pipe = dataio.pipeline(dataio.slot("payload").decode_bytes(), dataio.batch(2))
with pipe.bind(records).run("dlpack") as loader:
    for batch in loader:
        print(batch.keys, batch["payload"].shape)  # ['a', 'b'] [2, 3]

output= selects the framework. The loader's run() defaults to "torch" and also takes "numpy", "jax", "dlpack", a name registered with register_output_adapter, or a callable. Readers default to "numpy" (read_dcp to "torch"): read_image takes the loader's choices, and tensor readers such as read_safetensors take "numpy" or "torch". Column-batch iterators (iter_batches, iter_arrow_batches) yield {column: [values]} unless output="numpy". The examples are complete scripts.

Build a pipeline

A pipeline is optional record maps, one or more slot chains, one batch stage (batch, collate or group_by), then optional batch transforms and maps. Building it does no IO; .bind(records).run(...) starts a loader.

pipe = dataio.pipeline(
    dataio.map(lambda row: {"image": row["path"], "label": row["class_id"]}),
    dataio.slot("image").decode_image(mode="rgb").resize_short(size=256)
        .center_crop(h=224, w=224).normalize("imagenet"),
    dataio.slot("label").passthrough(),
    dataio.batch(64, drop_last=True, prefetch=4),
)
with pipe.bind(records).run("torch", concurrency=64) as loader:
    for batch in loader:
        train_step(batch["image"], batch["label"])

Slot methods are shorthand for the factories in these namespaces:

Namespace Operations
dataio.decoders Image, audio (resampling="sinc" anti-aliases), JSON, NPY, safetensors, text, bytes and PT decoding
dataio.transforms Resize, crop, normalize, augment and conditional transforms
dataio.batch_transforms MixUp, CutMix, padding, shuffle and normalization
dataio.cond Field and shape predicates for slot activation and transforms

A dataio.map before the batch stage gets input records, one after it the decoded Batch: a passthrough slot plus dataio.map(lambda b: collate_fn(b["sample"])) reuses an existing collator. collate() keeps the batches a source yields, group_by(key) batches each run of equal keys, and from_sampler(batch_sampler, row_fn) tags a batch sampler's rows for group_by("batch_id"). Examples: batch boundaries, conditional slots, conditional transforms; contracts: callbacks.

Choose sources

bind takes a reusable iterable or a zero-argument factory and reads it once per epoch; from_iterable(...) is single-pass. Wrap a source to index, compose, shuffle or mix it:

photos = dataio.records(dataset, indexed=True)  # len() + integer __getitem__
art = dataio.webdataset(manifest="s3://b/art/manifest.json")
records = dataio.mix([photos, art], [0.7, 0.3], seed=0, stop="first")
Call Effect
records(dataset, indexed=True) Integer row access, bypassing the dataset's iterator; with batch(N), __getitems__(indices) fetches rows together
.map_records(fn), .flat_map_records(fn), .concat(*sources) Replace one record, emit zero or more, append sources without opening them early
.shuffle(seed=42, buffer_size=4096) Mix nearby records in a buffer of that many records; no buffer_size means a full permutation
mix(sources, weights, seed=, stop=) Interleave indexed sources by weight into an indexed source; stop="first" ends before a source repeats, "all" once every source was read, "never" after length= samples
webdataset("shard-{0000..0099}.tar"), webdataset(manifest=...) Tar shards; a shard_writer manifest opens as an indexed, seekable source

Expanded and streaming sources need a buffered shuffle. Contracts: record sources, bulk reads, weighted mix; examples: callable sources, composition.

Configure execution

Loader options and process-wide environment settings bound concurrency and memory; dataio.env_vars() lists every setting with its default.

Setting Effect
run(..., concurrency=N) Per-shard stage admission; start with the default "auto", measure higher values for high-latency sources
batch(..., prefetch=K) Batches in flight
run(..., max_prefetch_bytes=B) Logical bytes of pending native outputs with a fixed layout; not an RSS limit
run(..., map_workers=N) Threads for record maps, sized-source __getitem__ and from_sampler record functions; output equals N=1, callbacks must be thread-safe
run(..., order="completion") Deliver batches as they finish instead of in submit order; no mid-epoch resume
DATAIO_CPU_WORKERS / DATAIO_IO_WORKERS Process-wide CPU / async IO worker pools
DATAIO_HTTP_MAX_INFLIGHT_READS Process-wide GET/HEAD limit; 0 means unlimited
DATAIO_OBJECT_SHARDS HTTP and object-store client shard fanout

From async code, run async for batch in batches inside async with loader.aiter() as batches: it reuses native prefetch and closes on early exit. One iteration owns a loader at a time. Scheduling, memory bounds and cleanup: plans and execution, iteration and cancellation.

Handle errors

run(..., on_failure=...) decides what a failed row does; batch.errors lists each failure (index, key, stage, op, message).

on_failure= Effect
"skip" (default) Drop failed rows; raise when no row of a batch survives
"strict" Raise on a failed row
N Drop failed rows; raise when fewer than N rows of a batch survive

log_errors=True logs each failure to the dataio.rows logger, or pass a callable. Source-iteration and Python batch-map errors propagate as raised (callback contracts).

Shard and resume

shard= splits each epoch across ranks; state_dict() / load_state_dict() checkpoint a loader mid-epoch. A keyed shuffle reshuffles a huge indexed dataset every epoch without materializing its index, and restores on any world size:

records = dataio.records(dataset, indexed=True, resume="seek")
loader = pipe.bind(records).run(
    "torch", shard={"id": rank, "of": world_size}, seed=42, shuffle="keyed",
)  # or shuffle=("block", 1024, 8) to read 1024-record blocks within 8-block windows
states = [None] * world_size
torch.distributed.all_gather_object(states, loader.state_dict())  # save `states`
# After restarting on any world size, every rank restores the gathered list:
loader.load_state_dict(states)

Ranks save after the same step; on an unchanged layout a rank can restore its own state_dict() alone. "pad": True gives every rank the same batch count. A restore replays the consumed prefix unless resume="seek" lets an indexed source with index-determined lookups skip it. Contracts: resume, checkpoint identity, elastic resume.

Read and write objects

Local paths, s3://, r2://, gs://, az://, hf:// and HTTP(S) URIs share these operations where the backend supports them:

dataio.configure_endpoint("r2", "https://<account>.r2.cloudflarestorage.com")
dataio.configure_transport("https", timeout_secs=30, readall_concurrency=32)
chunk = dataio.read("s3://bucket/file.bin", offset=0, size=1 << 20)
parts = dataio.fetch("s3://bucket/file.bin", ranges=[(0, 64), (4096, 256)])  # one GET
dataio.write("s3://bucket/out.bin", chunk, if_none_match="*")  # create only
etag = dataio.copy("s3://bucket/out.bin", "s3://bucket/copy.bin")
dataio.exists("s3://bucket/out.bin")  # False only when missing; other errors raise
dataio.ls("s3://bucket/prefix/", recursive=False, detail=True)  # one level, size/etag/mtime
dataio.glob("s3://bucket/**/*.safetensors")
payload = dataio.reader("s3://bucket/shard.tar.gz").gunzip().tar_entry("0001.jpg").read_all()
with dataio.writer("s3://bucket/checkpoint.bin") as writer:
    writer.write_chunks(huge_generator())
with dataio.shard_writer("s3://bucket/out/shard-{:06d}.tar", max_bytes=1 << 30) as shards:
    for key, jpg, meta in samples():
        shards.write({"__key__": key, "jpg": jpg, "json": meta})
records = dataio.webdataset(manifest=shards.manifest)  # indexed; no tar headers read

fetch_into(uri, buffer) fills an existing buffer, stream(uri) reads a body on demand (bytes, lines or server-sent events), head_many(uris) sends HEADs concurrently, and copy(recursive=True), sync and delete(recursive=True) work on trees like the command line. Requests retry throttling, 5xx, resets and timeouts; max_attempts= and deadline_secs= bound one call. On s3:// and r2://, the store verifies every upload's CRC64NVME, and every whole-object download is checked against the stored one.

s3:// uses the AWS credential chain (environment, profiles, SSO, web identity, ECS/EKS Pod Identity, instance metadata); configure_credentials(scheme, ...) sets explicit credentials, and clear_credentials(scheme) re-reads them. Details: network configuration, byte and format IO.

Command line

python -m dataio runs the same operations with the same credentials, so shell scripts need no other S3 tool; python -m dataio <command> --help lists its flags.

python -m dataio cp model.safetensors s3://bucket/models/     # -> .../models/model.safetensors
python -m dataio cp -r --workers 32 s3://bucket/models/vae /nvme/vae
python -m dataio cp -n 's3://bucket/ckpt/__3_*.distcp' /nvme/ckpt/
python -m dataio sync --workers 128 --delete --no-follow-symlinks /fsx/run s3://bucket/archive/run
python -m dataio rm -r s3://bucket/scratch/run-1              # prints "deleted 1234"
python -m dataio sync --dry-run --delete --exclude '*.tmp' /fsx/run s3://bucket/run   # plan only
python -m dataio --json sync s3://bucket/data /nvme/data      # {"copied": .., "skipped": .., "deleted": ..}
python -m dataio head --max-attempts 1 --deadline 5 s3://bucket/ckpt/COMMIT   # a fast probe
Command or flag Effect
cp SRC DST Copy a file or object (into DST/<name> when DST ends in / or is a local directory), a directory or prefix with -r, or every match of a wildcard
sync SRC DST Copy the files of a directory or prefix that are missing or differ under DST
rm URI Delete a file or object, or a directory or prefix with -r
head URI, cat URI, ls URI Print metadata (exit 1 when missing), the body (-o FILE saves it), or the URIs below a prefix (--long adds sizes)
info (the default), parquet PATH, safetensors PATH Print host capabilities, Parquet metadata or safetensors entries
cp -n Skip destinations that exist
cp --etag ETAG Copy one object only if it still has this ETag
sync --delete Remove destination files the source lacks
sync --strategy S size (default) compares sizes; mtime also copies files modified after their copy, between local trees; checksum also compares equal-size files by CRC64NVME (s3, r2, local)
sync --no-follow-symlinks Skip symlinks in a local source
--include PATTERN, --exclude PATTERN (cp -r or a wildcard, sync, rm -r) Select files by their path relative to the source: an include pattern, if any, must match and no exclude pattern may (* also matches /; repeatable)
--dry-run (cp, sync, rm) Print each planned copy SRC DST / delete URI and the summary; change nothing
cp - DST Stream stdin to DST (-n: create only)
--workers N (cp, sync) Transfers in flight (default 64)
--max-attempts N (not info, parquet, safetensors) Attempts per request on throttling, 5xx, resets and timeouts (default DATAIO_OBJECT_MAX_ATTEMPTS, 4)
--deadline SECS (likewise) One bound for the whole command: listing, transfers, retries and backoff
--json (before the command) Print JSON instead of text

Output. cp, sync and rm -r print one summary line (copied 3, skipped 1), or one JSON object with --json, and nothing per file. A failure prints one stderr line and exits 1; the first failure stops the remaining transfers, and re-running sync resumes. A usage error exits 2. Printed URLs drop userinfo and query strings, so a presigned signature never reaches a terminal or log.

Recursive commands treat a prefix as a directory (s3://b/run never reaches s3://b/run2/), and sync refuses a source with no file, so a mistyped source never empties the destination under --delete. Wildcards, filters, dry runs, path checks and retry limits: the command-line contract.

Replacing s5cmd 2.2.2:

s5cmd dataio
cp 's3://b/p/*' dir/, cp 'dir/*' s3://b/p/ cp -r s3://b/p dir, cp -r dir s3://b/p (local hidden files included)
cp 's3://b/p/*.parquet' dir/, cp -n 's3://b/p/__3_*' dir/ the same pattern (-n as well)
cp FILE s3://b/p/, cp s3://b/k /tmp/f the same
sync 's3://b/p/*' dir/, sync dir/ s3://b/p/ sync s3://b/p dir, sync dir s3://b/p (no wildcards)
--numworkers N --workers N (default 64)
--retry-count N --max-attempts N+1 (per command; default 4 attempts)
timeout SECS s5cmd ... --deadline SECS (exit 1 with a TimeoutError line)
sync --size-only default --strategy size
sync --size-only, then an ls count/bytes check sync --strategy checksum: exit 0 means every file's content matches
sync (size + newer mtime), local trees --strategy mtime
sync --delete, --no-follow-symlinks the same flags
--exclude P, --include P (cp, sync, rm) the same flags (paths relative to the source; * also matches /)
--dry-run the same flag (copy / delete lines, then the summary line)
--exit-on-error always
--log error, --stat default: no per-file lines, one summary line
--concurrency N, --part-size MiB (per-file parts) DATAIO_MULTIPART_CONCURRENCY (upload parts in flight per process, 64) / DATAIO_READALL_CONCURRENCY (ranges per download, 32), DATAIO_MULTIPART_PART_MIB (16; parts grow past it to fit 10 000) / DATAIO_READALL_CHUNK_MAX_MIB
rm 's3://b/p/*', rm 's3://b/p/*.tmp' rm -r s3://b/p, rm -r --include '*.tmp' s3://b/p
ls 's3://b/p/*' (count, bytes) --json ls --long s3://b/p/ (count, bytes)
cat s3://b/k cat s3://b/k
pipe DST cp - DST (-n: create only)
ls s3://b/k as an existence test head s3://b/k (exit 1 when missing), or dataio.exists in Python
run FILE (a batch of commands) not a command: call dataio.copy from Python, or cp -r / a wildcard
credential_process shim for Pod Identity not needed

Read structured formats

Typed readers return arrays, images, columns and Arrow data without a pipeline:

array = dataio.read_numpy("/data/x.npy")
image = dataio.read_image("s3://bucket/image.png", mode="rgb")
columns = dataio.read_parquet("/data/shard.parquet", columns=["key", "height"])
tall = dataio.scan_parquet("/data/shard.parquet").filter(lambda r: r["height"] > 512).collect()
stream = dataio.scan_feather("/data/table.arrow").select(["key"]).collect_arrow_stream()
for batch in dataio.scan_parquet("/data/features.parquet").iter_batches(output="numpy"):
    features, tokens = batch["features"], batch["tokens"]  # ndarray, RaggedArray
for example in dataio.iter_tfrecord("/data/train.tfrecord"):
    label = example["label"]

Scans keep Arrow types (collect_arrow_stream(), or PyArrow through collect_arrow_reader() / collect_arrow()). iter_batches(output="numpy") needs only NumPy: structs become dicts, fixed-size lists add dimensions, and variable-length lists become RaggedArray(values, offsets) (row i is values[offsets[i]:offsets[i + 1]]). iter_arrow_batches(source, output="numpy") does the same for any __arrow_c_stream__ producer, such as a DuckDB relation. Types, nulls and views: columnar NumPy batches.

Read or fill safetensors weights

Single-file and sharded checkpoints use the same API:

dataio.write_safetensors("checkpoints/model/", state_dict, max_shard_size="5GB")
subset = dataio.read_safetensors("checkpoints/model/", keys=["layer.weight"], output="torch")
with dataio.open_safetensors("weights.safetensors") as source:
    tensor = source.read(keys="layer.weight", output="torch")["layer.weight"]
    source.read_into(destinations, chunk_bytes=16 << 20)

read_into checks every shape, dtype and layout, then fills existing NumPy arrays or CPU/CUDA Torch tensors (destinations maps names to them) in bounded chunks, without another full copy. On local GDS-ready storage, read_safetensors(..., output="torch", device="cuda:0") reads straight to the GPU, falling back to CPU with a warning; gds_available() and gds_info() report support. Views, copies and CUDA staging: buffer ownership.

Convert distributed checkpoints

scan_dcp reassembles a PyTorch distributed checkpoint (DCP) into safetensors; read_dcp returns its tensors:

dataio.scan_dcp("s3://bucket/checkpoint", keys="model.", bf16=True).rename(
    lambda name: name.removeprefix("model.")
).write("weights.safetensors", strategy="stream")

DCP metadata is a Python pickle: load only trusted checkpoints. Selection, streaming and layouts: distributed checkpoints.

Transfer with RDMA

dataio.rdma does one-sided reads and writes through libfabric, loaded on first use:

import dataio.rdma as rdma

with rdma.serve(bind="0.0.0.0:0", buffer_size=1 << 20) as server:
    with rdma.connect(server.local_addr) as client:
        client.write_bytes(0, b"hello rdma")
        assert client.read_bytes(0, 10) == b"hello rdma"

serve(bind=..., file=path) exposes file ranges; rdma.read_into fills caller-owned CPU or CUDA buffers. available() also succeeds over TCP software providers, so check info() or diagnose() for physical RDMA. GPUDirect RDMA needs an HMEM provider and libfabric API 1.9+ (RDMA transport).

API index

Task Entry points
Build a pipeline pipeline, slot, slots, chain, batch, collate, group_by
Extend processing map, custom_op, register_output_adapter
Bind records records, files, urls, manifest, from_callable, from_iterable, from_sampler, webdataset, archive, mix
Work with bytes read, fetch, fetch_into, write, copy, sync, head, head_many, stream, ls, glob, exists, delete
Compose IO reader, writer, shard_writer, scan, from_iterator
Work with tensors read_safetensors, open_safetensors, write_safetensors, scan_dcp, read_dcp, scan_pt, read_pt
Work with arrays/tables read_* / scan_* / write_* for numpy, npz, parquet, feather, tfrecord; iter_arrow_batches, read_image, iter_tfrecord
Configure IO configure_credentials, configure_endpoint, configure_transport, clear_credentials
Use GDS gds_open, gds_register_buffer, gds_read_into_tensor

The documentation index links the contracts, the benchmark guide and the roadmap.

Develop

uv run --no-sync --with maturin maturin develop --profile staging
cargo test
uv run --no-sync python -m pytest python/tests -q
uv run --no-sync --with maturin maturin build --profile staging --out dist  # a wheel

The toolchain is pinned in rust-toolchain.toml. Source builds omit RDMA: install libfabric headers and add --features rdma (published wheels include it). Under a free-threaded interpreter (uv venv --python 3.14t) the same commands build the cp314t extension; maturin build -i python3.11 python3.14t builds both wheels. The native IO, network, Arrow and format crates build without Python.

License

Apache-2.0. Bundled dependency licenses are in THIRD_PARTY_NOTICES.

Metadata

Release files for dataio-rs 0.5.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dataio-rs 0.5.4
File Size Uploaded Provenance Actions
dataio_rs-0.5.4.tar.gz 1.4 MB Without provenance Details for dataio_rs-0.5.4.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for dataio-rs 0.5.4
File Interpreter ABI Platform Actions
dataio_rs-0.5.4-cp314-cp314t-manylinux_2_28_x86_64.whl CPython 3.14 CPython 3.14 free-threading Linux glibc 2.28+ x86-64 Details for dataio_rs-0.5.4-cp314-cp314t-manylinux_2_28_x86_64.whl
dataio_rs-0.5.4-cp311-abi3-manylinux_2_28_x86_64.whl CPython 3.11 abi3 Linux glibc 2.28+ x86-64 Details for dataio_rs-0.5.4-cp311-abi3-manylinux_2_28_x86_64.whl

Total release size: 30.3 MB

Release files / dataio_rs-0.5.4.tar.gz

Download URL dataio_rs-0.5.4.tar.gz
Size 1.4 MB
Tags Source
SHA-256 checksum
How to use checksums
8a9c3d0719b482e0fce9e42c1fbd521a03106462fb5087228a673bbc4c487453
BLAKE2b-256 checksum
How to use checksums
98e6707d7fd9b5985f217004e312bfb9974edc55053cacafb242ba7ce8dc473b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.12

Release files / dataio_rs-0.5.4-cp314-cp314t-manylinux_2_28_x86_64.whl

Download URL dataio_rs-0.5.4-cp314-cp314t-manylinux_2_28_x86_64.whl
Size 14.4 MB
Tags CPython 3.14 CPython 3.14 free-threading Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
5bb654e2cbd519ccad1081eb7f2e378d78f51dab203bcc0f2aa47b052d7201b5
BLAKE2b-256 checksum
How to use checksums
6514f3a1fe294d291a7640a43165e4cdcff4acef182ce204fb3793490fb372a8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.12

Release files / dataio_rs-0.5.4-cp311-abi3-manylinux_2_28_x86_64.whl

Download URL dataio_rs-0.5.4-cp311-abi3-manylinux_2_28_x86_64.whl
Size 14.4 MB
Tags CPython 3.11 Linux glibc 2.28+ x86-64 abi3
SHA-256 checksum
How to use checksums
70bcee77b263e8259ec5ba715631870deebf63a6d120a95dbdc3abf4d4edfb4d
BLAKE2b-256 checksum
How to use checksums
5f7d74a4fd37f871bb918ac862e689320c707ce4f6c6e7f20122c3f16b2368da
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.12

Security for dataio-rs 0.5.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page