ATdataset
A streaming audio-text dataloader for PyTorch training, built on WebDataset.
Designed for large-scale speech recognition training with multi-dataset muxing, dynamic batching, bucketing, and on-the-fly augmentation.
Install
pip install atdataset
For development:
git clone https://github.com/pkufool/ATdataset.git
cd ATdataset
pip install -e ".[dev]"
Quick Start
from atdataset import ATDataloader
dl = ATDataloader(
datasets="data/tars/train.lst",
sample_rate=16000,
max_duration=600.0,
)
for batch in dl:
audio = batch["audio"] # (B, T), padded
audio_lens = batch["audio_lens"] # (B,)
feature = batch["feature"] # (B, T, F), padded
feature_lens = batch["feature_lens"] # (B,)
texts = batch["text"] # list of str
break
CLI Tools
After installation, the atdataset command provides two subcommands:
# Build tar shards from a TSV manifest
atdataset build --input train.tsv --output-dir data/tars --num-tars 64
# Generate a .lst file from existing tar shards
atdataset gen_lst --audio-pattern 'data/tars/audios/*.tar' \
--txt-dir data/tars/manifests --output data/tars/train.lst
Run atdataset build --help or atdataset gen_lst --help for full option details.
Documentation
- Design & Architecture — pipeline architecture, bucketing, muxing, epoch control
- Usage Guide — detailed examples for all features
- Data Format — manifest, tar shard, and lst file formats
License
Apache-2.0
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file atdataset-0.2.5.tar.gz.
File metadata
- Download URL: atdataset-0.2.5.tar.gz
- Upload date:
- Size: 43.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.0.1 CPython/3.10.20
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
06efe14b687cc05f540df626edd3e48bd96470b88fbbb80b38699010f42bd404
|
|
| MD5 |
6e769530d87efcaa7a6a8b99f5ba9c99
|
|
| BLAKE2b-256 |
dfd999fa8ecc9f3ebf818ed73293fd48db569ba8c874be02063f2a7640586b9e
|
File details
Details for the file atdataset-0.2.5-py3-none-any.whl.
File metadata
- Download URL: atdataset-0.2.5-py3-none-any.whl
- Upload date:
- Size: 42.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.0.1 CPython/3.10.20
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
79e9d40ff823386bf96d8755739b0a8d2de427f8567ef7cc8cbf8c74a3b4a7ab
|
|
| MD5 |
2b27987df09003dd6cd971cd52ac5ffc
|
|
| BLAKE2b-256 |
f1ad6920f393d40cef7ab94d8d9110f96da406c085ea7bfe69a9bab069ea9618
|
