speechonnxmetrics

Unified speech evaluation metrics: no-reference MOS, intrusive (reference-based) signal metrics, ASR-based text metrics, and speaker similarity, behind one small API that runs on numpy and onnxruntime only.

Why numpy + onnxruntime only

The entire runtime depends on just numpy, onnxruntime and huggingface_hub. There is no torch in the runtime path. Neural MOS predictors (UTMOS, DNSMOS, NISQA, SIGMOS) run as exported ONNX graphs, so the package installs small, imports fast, and runs the same on CPU everywhere. The export extra pulls in torch/onnx, but only for offline model conversion by maintainers, never at inference time.

Model weights download from public HuggingFace repos on first use and cache locally. import speechonnxmetrics touches neither the network nor onnxruntime.

Install

Not yet on PyPI. Install from git:

pip install git+https://github.com/TigreGotico/speechonnxmetrics

Once published, pip install speechonnxmetrics will work too.

Extras (all optional):

pip install "speechonnxmetrics[audio,speaker]"

Quickstart

import speechonnxmetrics as s

# No-reference neural MOS (downloads UTMOS weights on first call, then caches):
print(s.score("test/fixtures/audio/source.wav", ["utmos"]))
# -> {'utmos': 4.4115...}

# Intrusive metrics need a clean reference (pure numpy, no download):
print(s.score("test/fixtures/audio/facodec_aria.wav",
              ["stoi", "mcd", "si_sdr"],
              ref="test/fixtures/audio/source.wav"))
# -> {'stoi': 0.662..., 'mcd': 10.459..., 'si_sdr': -26.937...}

Dict-valued metrics flatten into the result. dnsmos becomes dnsmos.sig, dnsmos.bak, dnsmos.ovrl. nisqa becomes nisqa.mos, nisqa.noi, and more. See examples/: every script there runs against the bundled fixture audio.

Metrics

21 metrics live in one registry. score()/score_batch() dispatch the audio metrics. The text metrics compare strings and are called directly from speechonnxmetrics.asr. Enumerate everything at runtime with s.list_metrics() or speechonnxmetrics list.

No-reference MOS (audio, needs model download)

Intrusive / reference-based (audio, pure numpy, needs ref=)

ASR-based text metrics (text, pure numpy, string in/out)

Speaker similarity and verification

Not in the score() registry. Call these from speechonnxmetrics.speaker: speaker_similarity (cosine between speaker embeddings, needs the speaker extra), plus pure-numpy eer, min_dcf and equal_error_threshold over score/label arrays.

Sample-rate handling is automatic: the base resamples input to each model's native rate (UTMOS/DNSMOS 16 kHz, SIGMOS 48 kHz, STOI analysis at 10 kHz), and NISQA is rate-adaptive and never resamples. See docs/metrics.md for the per-metric detail and paper citations, and docs/models.md for the ONNX models and their licences.

GPU inference

The ONNX-backed metrics (the MOS predictors, speaker similarity) run on CPUExecutionProvider by default. To run them on GPU, install onnxruntime-gpu (instead of, or alongside, onnxruntime) with the matching CUDA/cuDNN setup, then pick one of two knobs:

• SPEECHONNXMETRICS_PROVIDERS — a comma-separated onnxruntime provider list, e.g. SPEECHONNXMETRICS_PROVIDERS=CUDAExecutionProvider,CPUExecutionProvider. This is the knob that matters for batch pipelines and the CLI, which have no providers= argument to thread through — set the env var once and every ONNX session in the process picks it up with no code change.
• providers= — pass it explicitly to score()/score_batch(), or to an OnnxMetric subclass's constructor, when a given call needs providers different from the process default. An explicit providers= always wins over the env var.

Either way, a provider that onnxruntime does not have built in (or that fails to initialize) is silently dropped rather than raised: the requested list is intersected with onnxruntime.get_available_providers(), and CPUExecutionProvider is always kept as the final fallback so scoring never crashes for lack of a GPU.

CLI

$ speechonnxmetrics --help
usage: speechonnxmetrics [-h] [--version] {score,list} ...

positional arguments:
  {score,list}
    score       score one or more audio files
    list        list available metrics

$ speechonnxmetrics score --help
usage: speechonnxmetrics score [-h] [--ref REF] --metrics METRICS [--sr SR]
                               [--json]
                               audio [audio ...]

positional arguments:
  audio              degraded audio file(s) to score

options:
  --ref REF          reference audio file (required for intrusive metrics)
  --metrics METRICS  comma-separated metric names
  --sr SR            sample rate hint for raw input
  --json             emit JSON instead of a table

Real invocations:

$ speechonnxmetrics score test/fixtures/audio/facodec_aria.wav \
      --ref test/fixtures/audio/source.wav --metrics stoi,mcd,si_sdr
audio                                 mcd                 si_sdr               stoi
test/fixtures/audio/facodec_aria.wav  10.459728433678961  -26.937894650414812  0.6620030195244008

$ speechonnxmetrics list
name         kind   intrusive  requires_download
cer          text   True       False
dnsmos       audio  False      True
...

Full reference in docs/cli.md.

Licence

The package itself is Apache-2.0. Model weights carry their own licences:

nisqa is the one caveat: its weights are NonCommercial. Every other metric is safe for commercial use. The package makes no choice for you. It exposes the metric and states the terms, and selecting it is your call. Full per-model breakdown in docs/models.md.

Models are grouped in the HuggingFace collection speechonnxmetrics models under the TigreGotico org.

Not provided (on purpose)

• PESQ: ITU-T P.862 licensing is incompatible with an open, pip-installable package, and neural MOS predictors supersede it. Use a dedicated PESQ package under your own licence review if you need it.
• UTMOSv2: its published score is an ensemble over five folds times five random 3 s crops, so any single-fold single-crop export would be a different estimator, not an approximation of the published numbers.

Docs

• docs/index.md: orientation
• docs/metrics.md: every metric family in detail
• docs/usage.md: score vs score_batch, custom metrics, caching
• docs/cli.md: CLI reference
• docs/models.md: ONNX models, HF repos and licences