scoreq-pytorch 0.1.0

Unofficial fairseq-free PyTorch implementation of SCOREQ

SCOREQ-PyTorch

About • Model Types • Usage • How To Reproduce • Credits • License • Citation

About

This is an unofficial fairseq-free implementation of the SCOREQ Speech Quality Assessment system proposed in SCOREQ: Speech Quality Assessment with Contrastive Regression.

The original implementation provides a fairseq-based PyTorch model and an ONNX variant. In practice, the fairseq dependency can be difficult to install with recent Python, PyTorch, and dependency versions. The ONNX variant avoids fairseq, but it can be less convenient for PyTorch-based research workflows and may be difficult to run with GPU acceleration on ARM/aarch64 systems.

Recent study from ICASSP 2026 highlights the high correlation of SCOREQ with subjective listening scores for neural codecs. Therefore, modern neural audio codec and TTS research benefits from an easy-to-install SCOREQ implementation.

We provide a fairseq-free implementation written directly in PyTorch that matches the original system using converted weights and reimplemented modules.

We also provide a TorchScript variant that can be loaded with only PyTorch, without installing this package.

The PyTorch and TorchScript versions are validated against the original implementation and produce matching scores.

[!NOTE] In contrast to the original implementation, we support batched audio assessment. However, we recommend running SCOREQ with batch size 1 to avoid metric shifts caused by padding. Batching can be used for faster evaluation when small padding-related score differences are acceptable.

Model Types

As in the original system, we support 4 types of SCOREQ, i.e., 2 audio domains and 2 modes.

Data domain (what kind of audio is evaluated):

• natural: used for audio that was created from a genuine human speech (Audio Codecs, VoIP, Telephony, Speech Enhancement, Audio Restoration).
• synthetic: used for audio that was synthesized by a machine (Text-to-Speech (TTS), Voice Conversion (VC), Generative Speech Models).

Mode (whether there is a reference audio to compare with):

• nr: no-reference mode. Assesses the quality of audio, the higher the better, without relying on any reference.
• ref: reference mode. Calculate the distance between provided and reference audio embeddings, the lower the better.

We refer the user to the original repository and paper for more details on model types.

Usage

You can install the repo as a package:

pip install scoreq-pytorch

Or from source:

git clone https://github.com/Blinorot/scoreq-pytorch.git
cd scoreq-pytorch
pip install -e .

The code requires:

Package	Version
Python	>=3.9
PyTorch	>=2.2.0
HuggingFace Hub	>=0.20

The TorchScript checkpoint was scripted with PyTorch 2.5.1. We have tested that it works on PyTorch 2.2.0, however, PyTorch >=2.5.1 is recommended for the TorchScript variant.

Then, you can run the model as follows:

import torchaudio
from scoreq_pytorch import SCOREQScoreTorch

device = "cpu" # set to "cuda" to use on GPU
data_domain = "natural" # or "synthetic"
mode = "nr" # or "ref"
scoreq = SCOREQScoreTorch(
  data_domain=data_domain,
  mode=mode,
  device=device
) # already in eval mode

# load an audio file, e.g. using torchaudio
test_audio_path = ... # path to an audio file
test_wav, sr = torchaudio.load(test_audio_path)

# convert to MONO 16 kHz
TARGET_SR = 16000
if test_wav.shape[0] != 1:
    test_wav = test_wav[0:1]
if sr != TARGET_SR:
    test_wav = torchaudio.functional.resample(test_wav, orig_freq=sr, new_freq=TARGET_SR)
# put on device
test_wav = test_wav.to(device)

# for "ref" mode, you need a reference audio
# same loading and pre-processing procedure
if mode == "ref":
    ref_wav = ...
else:
    ref_wav = None

# calculate the score
# accepts T, 1xT, Bx1xT
scoreq_score = scoreq.score(test_wav, ref_wav) # tensor of shape (batch_size,)

You can replace SCOREQScoreTorch with SCOREQScoreScripted to use the TorchScript variant instead. On first use, the package downloads converted SCOREQ weights from Hugging Face Hub and caches them locally using the Hugging Face cache.

For TorchScript, you can avoid downloading the package and use the model directly:

import torch
import torchaudio
import wget

data_domain = "natural" # or "synthetic"
mode = "nr" # or "ref"

# download scripted checkpoint, e.g. using wget
checkpoint_url = f"https://huggingface.co/Blinorot/SCOREQ-PyTorch/resolve/main/scoreq_{data_domain}_{mode}_scripted.pt"
checkpoint_path = ... # path to saved checkpoint
wget.download(checkpoint_url, checkpoint_path)

# load directly with torch.jit
device = "cpu" # set to "cuda" to use on GPU
scoreq = torch.jit.load(checkpoint_path, map_location=device)
scoreq.eval()

# load an audio file, e.g. using torchaudio
test_audio_path = ... # path to an audio file
test_wav, sr = torchaudio.load(test_audio_path)

# convert to MONO 16 kHz
TARGET_SR = 16000
if test_wav.shape[0] != 1:
    test_wav = test_wav[0:1]
if sr != TARGET_SR:
    test_wav = torchaudio.functional.resample(test_wav, orig_freq=sr, new_freq=TARGET_SR)
# put on device
test_wav = test_wav.to(device)

# for "ref" mode, you need a reference audio
# same loading and pre-processing procedure
if mode == "ref":
    ref_wav = ...
else:
    ref_wav = None

# calculate the score
# accepts T, 1xT, Bx1xT
with torch.no_grad():
    scoreq_score = scoreq(test_wav, ref_wav) # tensor of shape (batch_size,)

Notes

The model expects audio sampled at 16 kHz.

Accepted tensor shapes:

Shape	Meaning
(T,)	single mono test_waveform
(1, T)	single mono test_waveform with channel dimension
(B, 1, T)	batch of mono test_waveforms

The input should be a floating point PyTorch tensor. Stereo audio should be converted to mono before scoring. scoreq.score(test_wav) returns a tensor of shape (batch_size,), where each value is a predicted quality score.

For reference ref mode, a reference audio ref_wav must be provided: scoreq.score(test_wav, ref_wav).

Note that score() and forward() return the same values. The only difference is that score() is decorated with torch.no_grad() for convenient inference. Since the raw TorchScript module exposes forward(), it is called directly as scoreq(test_wav, ref_wav) rather than through the package wrapper's scoreq.score(test_wav, ref_wav).

Batch size 1 is recommended to avoid padding-related score shifts.

API classes:

Class	Description
SCOREQScoreTorch	PyTorch implementation using converted weights.
SCOREQScoreScripted	Wrapper around the TorchScript checkpoint.

How To Reproduce

To reproduce PyTorch and Scripted checkpoints and validate them against the original SCOREQ module, follow the steps below.

First, install all required packages in a new environment:

# Optional
conda create -n scoreq python=3.9.7
conda activate scoreq

pip install pip==22.0
pip install -r requirements.txt

Then, you need to export weights from the original SCOREQ checkpoint:

# add --private to save privately
python extract_state_dict.py --repo-id USERNAME/REPO_NAME_ON_HUGGINGFACE

This will upload the state dict extracted from the original SCOREQ checkpoint (for each data domain and mode pair) to Hugging Face. The same state dict is used to load our fairseq-free PyTorch-only module.

To create a scripted version of the PyTorch model that allows to load SCOREQ without class definitions, run

# add --private to save privately
python create_scripted_model.py --repo-id USERNAME/REPO_NAME_ON_HUGGINGFACE

It will upload the scripted model to HuggingFace as well.

Finally, to test that all 3 variations (Original, PyTorch, Scripted) return the same scores, run

# set --device "cpu" to run on cpu
# set --batch-size to a value bigger than 1 to test batched version
python test.py --device "cuda" --batch-size 1

The models are tested on test-clean partition of LibriSpeech against DAC-regenerated codec audio for natural mode and MMS-TTS-synthesized TTS audio for synthetic mode.

Data Domain	Mode	SCOREQ Version	Score (LibriSpeech Test-Clean)
natural	nr	Original	4.184167324040683
natural	nr	Torch	4.184167324040683
natural	nr	Scripted	4.184167324040683
natural	ref	Original	0.10563717598792251
natural	ref	Torch	0.10563717685383922
natural	ref	Scripted	0.10563717685383922
synthetic	nr	Original	4.465621592707306
synthetic	nr	Torch	4.465621592707306
synthetic	nr	Scripted	4.465621592707306
synthetic	ref	Original	0.5238628057009391
synthetic	ref	Torch	0.5238628115362793
synthetic	ref	Scripted	0.5238628115362793

Credits

The code is based on the original SCOREQ and fairseq repositories.

License

This project is released under the MIT License.

Parts of the implementation are adapted from the original SCOREQ and fairseq repositories, which are also MIT licensed. See LICENSES for third-party license texts.

Converted checkpoints are derived from the original SCOREQ checkpoint. Original authors retain copyright over the original model and weights.

Citation

If you use this package, please cite the original SCOREQ paper:

@article{ragano2024scoreq,
  title={SCOREQ: Speech quality assessment with contrastive regression},
  author={Ragano, Alessandro and Skoglund, Jan and Hines, Andrew},
  journal={Advances in Neural Information Processing Systems},
  volume={37},
  pages={105702--105729},
  year={2024}
}