mavdt 0.1.0

Manifestro Audio Voice Dataset Toolkit — download, subset, and clean Mozilla Common Voice data

MAVDT

Manifestro Audio Voice Dataset Toolkit

Download, subset, and clean Mozilla Common Voice datasets — from 3-speaker test sets to 50+ hour corpora.

---

Overview

MAVDT is a modular CLI + Python toolkit that streamlines working with Mozilla Common Voice data. It covers the full preprocessing pipeline in three composable steps:

MDC API  ──►  mavdt-download  ──►  mavdt-select  ──►  mavdt-clean  ──►  Training-ready WAVs

Each step produces a manifest (manifest.jsonl or manifest.csv) so you always know exactly what's in your dataset and why.

---

Features

	Capability	Details
⬇️	Download	Fetch datasets via Mozilla Data Collective API with streaming and progress bars
✂️	Subset	Four selection modes — by speakers, all records, by duration, or random sample
🔊	Clean	Resample, mono-convert, peak/RMS normalize, filter by duration and loudness
📋	Manifests	Output manifest.jsonl and/or manifest.csv with full per-clip metadata
🔁	Reproducible	Seeded random selection, YAML config files, deterministic pipelines
🐍	Dual interface	CLI scripts and importable Python API — use in shell or Jupyter notebooks

---

Quick Start

1. Install

git clone https://github.com/manifestro/audio-dataset-toolkit.git
cd audio-dataset-toolkit
pip install -e .

2. Set your API key

export MAVDT_API_KEY="your_mdc_api_key"

Get your key at Mozilla Data Collective.

3. Run the pipeline

# Download dataset
mavdt-download --dataset_id cmndapwry02jnmh07dyo46mot --output_dir data/raw

# Select 3 female speakers, 5 clips each
mavdt-select \
  --input_dir data/raw/kk \
  --mode speakers \
  --num_speakers 3 \
  --max_clips_per_speaker 5 \
  --gender female \
  --output_dir data/selected

# Clean: 16 kHz · mono · peak-normalized
mavdt-clean \
  --input_dir data/selected \
  --output_dir data/cleaned \
  --target_sr 16000

That's it — data/cleaned/ now contains ready-to-train WAV files and a manifest.jsonl.

---

Installation

# Core install
pip install -e .

# With dev tools (pytest, ruff, mypy)
pip install -e ".[dev]"

# With notebook visualisation support
pip install -e ".[viz]"

Requirements: Python ≥ 3.11

---

CLI Reference

mavdt-download

Fetch a Common Voice dataset from the Mozilla Data Collective.

mavdt-download --dataset_id ID [--api_key KEY] [--output_dir DIR] [--force] [--config FILE]

Flag	Default	Description
--dataset_id	required	MDC dataset identifier
--api_key	$MAVDT_API_KEY	API Bearer token
--output_dir	data/raw	Destination directory
--force	false	Overwrite existing files
--remove_archive	true	Delete .tar.gz after extraction
--config	—	Path to YAML config file

---

mavdt-select

Subset a downloaded dataset using different selection strategies.

mavdt-select --input_dir DIR --output_dir DIR [options]

Flag	Default	Description
--input_dir	required	Extracted dataset directory
--output_dir	required	Output directory
--mode	speakers	speakers · all · length · random
--dataset_file	validated.tsv	Source TSV file
--num_speakers	—	Number of speakers (mode=speakers)
--max_clips_per_speaker	—	Max clips per speaker
--min_clips_per_speaker	1	Min clips a speaker must have to qualify
--max_total_clips	—	Global clip limit (mode=all / random)
--max_total_hours	—	Global hours limit (mode=all)
--min_duration_sec	—	Min duration filter
--max_duration_sec	—	Max duration filter
--gender	—	male or female filter
--flat	false	Flat output (no per-speaker subdirs)
--format	jsonl	jsonl · csv · both
--seed	42	Random seed
--config	—	Path to YAML config file

Selection modes

# speakers — N speakers, controlled clips per speaker
mavdt-select --mode speakers --num_speakers 10 --max_clips_per_speaker 20 --gender male

# all — every record, capped by hours
mavdt-select --mode all --max_total_hours 50

# length — only clips within a duration window
mavdt-select --mode length --min_duration_sec 2.0 --max_duration_sec 10.0

# random — N random clips, no speaker grouping
mavdt-select --mode random --max_total_clips 1000 --seed 7

---

mavdt-clean

Resample, normalize, and filter audio clips.

mavdt-clean --input_dir DIR --output_dir DIR [options]

Flag	Default	Description
--input_dir	required	Directory with manifest + audio
--output_dir	required	Output directory
--target_sr	16000	Target sample rate (Hz)
--mono / --no-mono	mono	Convert to mono
--normalize	peak	peak · rms · none
--target_rms	0.1	Target RMS value for rms normalization
--min_duration_sec	1.0	Drop clips shorter than this
--max_duration_sec	30.0	Drop clips longer than this
--min_rms_db	-40.0	Drop clips quieter than this (dB)
--format	jsonl	jsonl · csv · both
--config	—	Path to YAML config file

---

Configuration File

Every flag can be set in a YAML config. CLI arguments take precedence over file values.

# examples/config_example.yaml

download:
  api_key: "YOUR_MDC_API_KEY"       # or set $MAVDT_API_KEY
  dataset_id: "cmndapwry02jnmh07dyo46mot"
  output_dir: "data/raw"
  force: false

select:
  input_dir: "data/raw/kk"
  mode: "speakers"
  output_dir: "data/selected"
  num_speakers: 3
  max_clips_per_speaker: 5
  gender: "female"
  seed: 42

clean:
  input_dir: "data/selected"
  output_dir: "data/cleaned"
  target_sr: 16000
  normalize: "peak"
  min_duration_sec: 1.0
  max_duration_sec: 30.0
  min_rms_db: -40.0

Pass it to any command:

mavdt-download --config examples/config_example.yaml
mavdt-select   --config examples/config_example.yaml
mavdt-clean    --config examples/config_example.yaml

---

Python API

Use mavdt directly in scripts or Jupyter notebooks:

from mavdt.download import download_dataset
from mavdt.select import run_selection
from mavdt.clean import clean_dataset

# Download
dataset_path = download_dataset(
    dataset_id="cmndapwry02jnmh07dyo46mot",
    api_key="YOUR_KEY",
    output_dir="data/raw",
)

# Subset — 3 female speakers, 5 clips each
selected_path = run_selection(
    input_dir="data/raw/kk",
    output_dir="data/selected",
    mode="speakers",
    num_speakers=3,
    max_clips_per_speaker=5,
    gender="female",
    seed=42,
)

# Clean — 16 kHz mono, peak-normalized, drop quiet clips
cleaned_path = clean_dataset(
    input_dir="data/selected",
    output_dir="data/cleaned",
    target_sr=16000,
    normalize="peak",
    min_rms_db=-40.0,
)

See notebooks/mavdt_examples.ipynb for a full walkthrough with visualisations.

---

Output Format

Every mavdt-select and mavdt-clean run produces audio files alongside a manifest.

Directory layout

data/selected/
├── manifest.jsonl          # one JSON object per clip
├── speaker_001/
│   ├── common_voice_kk_1234.mp3
│   └── common_voice_kk_1235.mp3
├── speaker_002/
│   └── common_voice_kk_1236.mp3
└── ...

With --flat, all clips land directly in the output directory with no subdirectories.

Manifest schema

// manifest.jsonl — one record per line
{
    "audio_path": "speaker_001/common_voice_kk_1234.wav",
    "client_id":  "a1b2c3d4e5f6",
    "sentence":   "Сәлем, әлем!",
    "duration":   3.45,
    "gender":     "female",
    "age":        "twenties",
    "up_votes":   2,
    "down_votes": 0,
    "locale":     "kk"
}

Field	Type	Description
audio_path	str	Relative path from the manifest directory
client_id	str	Speaker identifier (hashed by Common Voice)
sentence	str	Transcription text
duration	float	Duration in seconds
gender	str	male · female · nan
age	str	Age bracket from Common Voice
up_votes	int	Community up-vote count
down_votes	int	Community down-vote count
locale	str	Language code (kk, ru, en, …)

---

Project Structure

audio-dataset-toolkit/
├── src/mavdt/
│   ├── download.py         # MDC API download & tar.gz extraction
│   ├── select.py           # Subsetting, sampling, manifest writing
│   ├── clean.py            # Resample, normalize, filter
│   ├── utils.py            # Logging, statistics, manifest I/O
│   └── config.py           # Dataclasses & YAML config loader
├── scripts/
│   ├── download_commonvoice.py
│   ├── select_samples.py
│   └── clean_dataset.py
├── notebooks/
│   └── mavdt_examples.ipynb
├── examples/
│   └── config_example.yaml
├── docs/
│   ├── usage.md
│   ├── scripts.md
│   └── arguments.md
└── tests/
    ├── test_download.py
    ├── test_select.py
    ├── test_clean.py
    └── test_utils.py

---

Contributing

Contributions are welcome. Please read CONTRIBUTING.md before opening a pull request.

# Set up dev environment
pip install -e ".[dev]"

# Run tests
pytest

# Lint
ruff check src/ tests/

Planned extensions (good first issues):

• scripts/balance_dataset.py — balance speakers / gender
• scripts/split_train_val.py — train / val / test splits
• scripts/export_hf.py — push to HuggingFace Hub
• scripts/analyze_dataset.py — deep statistics & plots

---

License

MIT © 2026 Manifestro Inc.