xares 0.0.5

eXtensive Audio Representation and Evaluation Suite

X-ARES: eXtensive Audio Representation and Evaluation Suite

Introduction

X-ARES is a toolkit for training, evaluating, and exporting audio encoders for various audio tasks. It is heavily inspired by the HEAR benchmark.

Supported tasks

Speech

• Speech Commands V2
• LibriCount
• VoxLingua107
• VoxCeleb1
• LibriSpeech-Male-Female
• Fluent Speech Commands
• VocalSound
• CREMA-D
• RAVDESS
• LibriSpeech-Phoneme
• ASV2015

Environment

• ESC-50
• FSD50k
• UrbanSound 8k
• DESED
• FSD18-Kaggle
• Clotho

Music

• MAESTRO
• GTZAN Genre
• NSynth
• FMA

Installation

X-ARES is available on PyPI. You can install it via pip.

pip install xares

For development, you can clone the repository and install the package in editable mode.

git clone <this-repo>
cd xares
pip install -e .[examples]

Run with the baseline pretrained audio encoder (Dasheng)

You can run the benchmark with the baseline pretrained audio encoder (Dasheng) with 8 parallel jobs using the following command:

python -m xares.run --max-jobs 8 example/dasheng/dasheng_encoder.py src/tasks/*.py

It will download the datasets from Zenodo, and then evaluate the encoder on all the tasks. If the automatic download fails, you can also manually download the datasets using tools/download_manually.sh.

Alternatively, you can run tasks from within Python. Here is an example of running the ASVspoof2015 task in a single process:

>>> from example.dasheng.dasheng_encoder import DashengEncoder
>>> from tasks.asvspoof_task import asvspoof2015_config
>>> from xares.task import XaresTask

>>> task = XaresTask(encoder=DashengEncoder(), config=asvspoof2015_config())
>>> task.run()

Run with your own pretrained audio encoder

Two examples of audio encoder wrapper could be found at example/dasheng/dasheng_encoder.py and example/wav2vec2/wav2vec2.py.

We provide a check function to verify if the encoder is correctly implemented:

>>> from xares.audio_encoder_checker import check_audio_encoder

>>> encoder = YourEncoder()
>>> check_audio_encoder(encoder)
True

And then you can run the benchmark with your own encoder:

python -m xares.run --max-jobs 8 your_encoder.py src/tasks/*.py

Add new tasks

Adding a new task is easy. Refer to the existing task implementations for guidance. You need to create a TaskConfig tailored to your chosen dataset.