Data preparation

First, install the discophon.prepare package:

pip install discophon.prepare

You also need the sox binary available in your $PATH for pre-processing audio files.

Now let's say you want to install the benchmark data and assets in a directory $DATA.

Download Common Voice data

You first need to download audio data from CommonVoice. You can use their API if you don't want to download large files on your local computer.

Download everything in $DATA/raw.

Dev languages:

• Common Voice Scripted Speech 23.0 - German (34.41 GB)
• Common Voice Scripted Speech 23.0 - Swahili (21.23 GB)
• Common Voice Scripted Speech 23.0 - Tamil (8.56 GB)
• Common Voice Scripted Speech 23.0 - Thai (8.35 GB)
• Common Voice Scripted Speech 23.0 - Turkish (2.73 GB)
• Common Voice Scripted Speech 23.0 - Ukrainian (2.55 GB)

Test languages:

• Common Voice Scripted Speech 23.0 - Basque (14.58 GB)
• Common Voice Scripted Speech 23.0 - Chinese (China) (21.26 GB)
• Common Voice Scripted Speech 23.0 - English (86.83 GB)
• Common Voice Scripted Speech 23.0 - French (27.87 GB)
• Common Voice Scripted Speech 23.0 - Japanese (11.80 GB)
• Wolof data comes from a different source, and will be downloaded with the other assets in the following section.

Extract each archive, with tar --strip-components=1 -xvf ....

For example, let's say your archive is named mcv-scripted-uk-v23.0.tar.gz. Extract it with tar --strip-components=1 -xvf mcv-scripted-uk-v23.0.tar.gz, and move the output directory to $DATA/raw.

You can delete the archives afterwards. You should have the following structure:

❯ tree -L 2 $DATA
$DATA
└── raw
    ├── de
    ├── en
    ├── eu
    ├── fr
    ├── ja
    ├── sw
    ├── ta
    ├── th
    ├── tr
    ├── uk
    └── zh-CN

Download benchmark assets

Now download the benchmark assets with the following command:

python -m discophon.prepare download $DATA

This will download:

• Symlinks to audio files for each split in each language
• Manifests
• Alignments and item files

Preprocess selected audio files

Now resample audio files and convert them to WAV with the command:

for code in de en eu fr ja sw ta th tr uk zh-CN; do
    python -m discophon.prepare audio $DATA $code
done

This will create directories $DATA/audio/cmn/all, $DATA/audio/deu/all, $DATA/audio/eng/all, etc. with resampled audio files. The directories corresponding to each split contain symlinks to those files.

You can parellize this loop. You can delete the $DATA/raw folder afterwards.

If you are in a SLURM cluster, you should also parallelize each dataset processing across tasks or array jobs. The discohpon.prepare package will automatically handle the distribution of files to process across jobs.