ATdataset
A streaming audio-text dataloader for PyTorch training, built on WebDataset.
Designed for large-scale speech recognition training with multi-dataset muxing, dynamic batching, bucketing, and on-the-fly augmentation.
Install
pip install atdataset
For development:
git clone https://github.com/pkufool/ATdataset.git
cd ATdataset
pip install -e ".[dev]"
Quick Start
from atdataset import ATDataloader
dl = ATDataloader(
datasets="data/tars/train.lst",
sample_rate=16000,
max_duration=600.0,
)
for batch in dl:
audio = batch["audio"] # (B, T), padded
audio_lens = batch["audio_lens"] # (B,)
feature = batch["feature"] # (B, T, F), padded
feature_lens = batch["feature_lens"] # (B,)
texts = batch["text"] # list of str
break
CLI Tools
After installation, the atdataset command provides two subcommands:
# Build tar shards from a TSV manifest
atdataset build --input train.tsv --output-dir data/tars --num-tars 64
# Generate a .lst file from existing tar shards
atdataset gen_lst --audio-pattern 'data/tars/audios/*.tar' \
--txt-dir data/tars/manifests --output data/tars/train.lst
Run atdataset build --help or atdataset gen_lst --help for full option details.
Documentation
- Design & Architecture — pipeline architecture, bucketing, muxing, epoch control
- Usage Guide — detailed examples for all features
- Data Format — manifest, tar shard, and lst file formats
License
Apache-2.0
1 maintainer
Metadata
1 maintainer
Release files for atdataset 0.2.6
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| atdataset-0.2.6.tar.gz | 43.3 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| atdataset-0.2.6-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 85.5 kB
Release files / atdataset-0.2.6.tar.gz
| Download URL | atdataset-0.2.6.tar.gz |
|---|---|
| Size | 43.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2a738fc7e047ea9b36e3af94d3b41f597b9288fa24f2ae2003e6508398ebfcdd
|
|
BLAKE2b-256 checksum How to use checksums |
a164fb2d298b44e35935e4935fb12274d0f703c6fc81f9aae9765087168e3478
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.0.1 CPython/3.10.20
|
Release files / atdataset-0.2.6-py3-none-any.whl
| Download URL | atdataset-0.2.6-py3-none-any.whl |
|---|---|
| Size | 42.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2c43bfd4d0cc8a0c4a78f96e57ea3199a31fdfe2308519e3cfb54efaae776cea
|
|
BLAKE2b-256 checksum How to use checksums |
8ffc825d438308614bee382dab231ff97d856a0374309c15a4ec3b59fa31964c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.0.1 CPython/3.10.20
|
Security for atdataset 0.2.6
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
