This repository is a fork of the original UTMOS v2 project, maintained by the Hyperion organization. It packages the original implementation under the distribution name hyperion-utmosv2 and adjusts dependency constraints so it can coexist with Hyperion. It is not an official UTMOS release and is not necessarily endorsed by the original authors.

The original utmosv2 Python import name is intentionally preserved:

import utmosv2

The original MIT license, copyright notice, paper citation, and attribution are preserved in this fork.

UTMOSv2: UTokyo-SaruLab MOS Prediction System

🎤✨ Original project's official implementation ✨🎤
“The T05 System for The VoiceMOS Challenge 2024:
Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech”
🏅🎉 accepted by IEEE Spoken Language Technology Workshop (SLT) 2024. 🎉🏅

ꔫ･-･ꔫ･-･ꔫ･-･ꔫ･-･ꔫ･-･ꔫ･-･ꔫ･-･ꔫ

✨  UTMOSv2 achieved 1st place in 7 out of 16 metrics  ✨
✨🏆    and 2nd place in the remaining 9 metrics    🏆✨
✨    in the VoiceMOS Challenge 2024 Track1!    ✨

🚀 Quick Prediction

✨ You can easily use the pretrained UTMOSv2 model!

🛠️ Using in your Python code 🛠️

If you want to make predictions using the UTMOSv2 library, follow these steps:

1. Install the Hyperion-maintained package from PyPI

   pip install hyperion-utmosv2
   

   This package can be installed in the same environment as Hyperion. Install Hyperion according to its own documentation, then install hyperion-utmosv2 in that environment.

2. Make predictions

   • To predict the MOS of a tensor or array already loaded in memory:

     import utmosv2
     
     model = utmosv2.create_model(pretrained=True)
     # data: torch.Tensor or np.ndarray with shape (batch_size, sequence_length) or (sequence_length,)
     # sr: Sampling rate of the input audio data. If not provided, it defaults to 16000 Hz.
     mos = model.predict(
         data=data, sr=16000
     )  # Returns a torch.Tensor or np.ndarray with shape (batch_size,) or (1,)
     

   • To predict the MOS of a single .wav file:

     import utmosv2
     
     model = utmosv2.create_model(pretrained=True)
     mos = model.predict(input_path="/path/to/wav/file.wav")  # Returns a float value
     

   • To predict the MOS of all .wav files in a folder:

     import utmosv2
     
     model = utmosv2.create_model(pretrained=True)
     mos = model.predict(
         input_dir="/path/to/wav/dir/"
     )  # Returns a list of dicts with 'file_path' and 'predicted_mos' keys
     

[!NOTE] When data is provided, input_path and input_dir are ignored.

[!NOTE] Either input_path or input_dir must be specified when data is None, but not both.

📜 Using the inference script 📜

If you want to make predictions using the inference script, follow these steps:

1. Clone this repository and navigate to UTMOSv2 folder

   git clone https://github.com/sarulab-speech/UTMOSv2.git
   cd UTMOSv2
   

2. Install Package

   pip install -e '.[optional]'
   

3. Make predictions

   • To predict the MOS of a single .wav file:

     python inference.py --input_path /path/to/wav/file.wav --out_path /path/to/output/file.csv
     

   • To predict the MOS of all .wav files in a folder:

     python inference.py --input_dir /path/to/wav/dir/ --out_path /path/to/output/file.csv
     

[!NOTE] If you are using zsh, quote the extras specifier like this:

pip install -e '.[optional]'

[!TIP] If --out_path is not specified, the prediction results will be output to the standard output. This is particularly useful when the number of files to be predicted is small.

[!NOTE] Either --input_path or --input_dir must be specified, but not both.

[!NOTE] These methods provide quick and simple predictions. For more accurate predictions and detailed usage of the inference script, please refer to the inference guide.

🤗 You can try a simple demonstration on Hugging Face Space:

Model files, external assets, and limitations

The Python package does not bundle model checkpoints, datasets, caches, or audio files. create_model(pretrained=True) downloads the pretrained checkpoint at runtime from the upstream UTMOSv2 Hugging Face repository and stores it in the local UTMOSv2 cache; network access is therefore required for the first pretrained-model use. A custom checkpoint can be supplied with checkpoint_path. The package also relies on external pretrained transformer assets when the selected configuration uses them.

Dataset licenses and usage terms vary by dataset; see docs/datasets.md and comply with each dataset's terms. The code is distributed under the MIT license in LICENSE, while pretrained weights, model cards, datasets, and third-party assets may have separate terms. Review those terms before redistribution or commercial use. Predictions are model estimates and may not be suitable as the sole basis for safety-critical, compliance, or other high-stakes decisions.

Publishing and release setup

The distribution name is hyperion-utmosv2; the import name remains utmosv2. The repository's release workflow builds and validates source and wheel distributions, then publishes only when a GitHub Release is published using PyPI Trusted Publishing/OIDC.

Before the first release:

1. Verify the PyPI project page is unavailable or belongs to this organization. If the name is taken, choose a different distribution name and update pyproject.toml, this README, and the Trusted Publisher configuration.

2. On PyPI, add a Trusted Publisher for owner hyperion-ml, repository UTMOSv2, workflow .github/workflows/python-publish.yml, and environment pypi.

3. In GitHub, create the pypi environment. Require approval from an appropriate maintainer and restrict deployments to the release workflow or protected release rules as appropriate for the organization. Do not add a PyPI API token; the workflow uses OIDC.

4. Merge the workflow to the repository's default main branch. Create a GitHub Release with a new tag (for example v1.3.1) and publish the release. A draft or prerelease does not trigger this workflow.

5. Verify the files, metadata, and installation from PyPI:

   python -m pip install --upgrade hyperion-utmosv2
   python -c "import utmosv2; print(utmosv2.__version__)"
   

   Also inspect the PyPI project page and python -m pip show hyperion-utmosv2. Use python -m build and twine check dist/* before release.

If hyperion-utmosv2 is already registered, do not attempt to claim or overwrite it. Confirm ownership and contact the current owner if appropriate, or select an available name and update all package and publishing references consistently.

⚒️ Train UTMOSv2 Yourself

If you want to train UTMOSv2 yourself, please refer to the training guide. To reproduce the training as described in the paper or used in the competition, please refer to this document.

📂 Used Datasets

Details of the datasets used in this project can be found in the datasets documentation.

🔖 Citation

If you find UTMOSv2 useful in your research, please cite the following paper:

@inproceedings{baba2024utmosv2,
  title     = {The T05 System for The {V}oice{MOS} {C}hallenge 2024: Transfer Learning from Deep Image Classifier to Naturalness {MOS} Prediction of High-Quality Synthetic Speech},
  author    = {Baba, Kaito and Nakata, Wataru and Saito, Yuki and Saruwatari, Hiroshi},
  booktitle = {IEEE Spoken Language Technology Workshop (SLT)},
  year      = {2024},
  pages     = {818--824},
  doi       = {10.1109/SLT61566.2024.10832315},
}

:octocat: GitHub Star History