hf-dataset-slow-disk

A Hugging Face-style Parquet dataset library for HDDs, network filesystems, and other slow storage.

Features

• Sequential row-group reads with bounded memory
• Lazy background download and audio decoding
• Standard Hugging Face Hub Parquet cache
• Resumable general metadata cache with visible progress logs
• Deterministic shuffle, epochs, and exact batch resume
• Sample-mixed concatenation with one shared worker pipeline

Install

pip install hf-dataset-slow-disk

Quick start

from hf_dataset_slow_disk import load_dataset

dataset = load_dataset(
    "capleaf/viVoice",
    split="train",
    batch_size=8,
    num_workers=2,
).shuffle(seed=42)

dataset.set_epoch(0)

try:
    for batch in dataset:
        train_step(batch)
finally:
    dataset.close()

Workers start lazily on the first get() or when dataset iteration begins. Defaults are one worker, an eight-batch decoded buffer, and a two-file Parquet buffer.

An optional process-pickleable collate_fn(batch) runs on decoded column-oriented batches inside each background worker. It must preserve column names, order, and row count, allowing CPU preprocessing to overlap model execution.

Core API

from hf_dataset_slow_disk import (
    Audio,
    ConcatDataset,
    Dataset,
    DatasetDict,
    concatenate_datasets,
    load_dataset,
)

• load_dataset() loads Hub or local Parquet data.
• shuffle(seed) and set_epoch(epoch) select deterministic order.
• set_batch_idx(batch_idx) restores the next training batch.
• for batch in dataset, get(), and iter_batches() consume batches.
• iter_rows() performs synchronous sample-by-sample iteration.
• concatenate_datasets() mixes compatible sources without oversampling.
• cast_column() configures lazy audio decoding.

Documentation

• Slow-disk runtime and caching
• Deterministic shuffle and resume
• Dataset concatenation

Examples

uv run python examples/basic.py
uv run python examples/benchmark.py
uv run python examples/concat.py

The benchmark and concat examples use batch size 8 and stop after 10 batches.

Disclaimer

This project was developed with the assistance of AI coding tools. Its source code is publicly available at giangndm/hf-dataset-slow-disk. Please review and validate it for your own workloads before production use.

Development

uv sync
uv run pytest -q
uv build