OpenSTBench

English | 中文

OpenSTBench is a multidimensional evaluation toolkit for speech translation. It is designed for heterogeneous systems, including speech-to-text translation (S2TT), speech-to-speech translation (S2ST), offline systems, and streaming systems.

The toolkit organizes evaluation into three dimensions:

• Translation Quality: whether the translated text preserves the source meaning.
• Speech Quality: whether generated speech is natural, text-consistent, speaker-preserving, emotion-preserving, and faithful to non-verbal or paralinguistic events.
• Temporal Quality: whether generated speech preserves duration structure and, for streaming systems, whether output is responsive.

Installation

pip install OpenSTBench

For local development:

git clone https://github.com/sjtuayj/OpenSTBench.git
cd OpenSTBench
conda create -n openstbench python=3.10 -y
conda activate openstbench
pip install -e .

Optional extras:

pip install "OpenSTBench[comet]"
pip install "OpenSTBench[whisper]"
pip install "OpenSTBench[speech_quality]"
pip install "OpenSTBench[emotion]"
pip install "OpenSTBench[paralinguistics]"
pip install "OpenSTBench[metricx]"
pip install "OpenSTBench[all]"

MetricX follows the official google-research/metricx runtime requirements. Installing OpenSTBench[metricx] or OpenSTBench[all] pins the MetricX-compatible stack, including transformers[torch]==4.30.2, sentencepiece==0.1.99, datasets==2.13.1, protobuf==3.20.3, and accelerate>=0.26.0.

BLEURT is installed separately:

pip install git+https://github.com/lucadiliello/bleurt-pytorch.git

Package Names

• PyPI package: OpenSTBench
• Python import: openstbench

Evaluation Dimensions

Offline and streaming are supported system settings, not separate metric dimensions. Use the evaluators that match the available outputs: text, generated speech, source/target audio pairs, event annotations, or streaming traces.

Experimental Overview

The radar plot below illustrates the multidimensional view produced by OpenSTBench for representative streaming and offline speech translation systems. It summarizes how systems can differ across translation quality, speech quality, and temporal quality: a system with strong translation quality may still show different behavior in speech realization, speaker or emotion preservation, paralinguistic fidelity, temporal consistency, and latency or efficiency.

Datasets

The paper uses the following datasets. Please follow the license and access terms of each original dataset.

Quick Start

from openstbench import TranslationEvaluator

evaluator = TranslationEvaluator(
    use_bleu=True,
    use_chrf=True,
    use_comet=False,
    use_bleurt=False,
    use_metricx=True,
    device="cuda",
)

scores = evaluator.evaluate_all(
    reference=["我喜欢看电影。", "今天天气很好。"],
    target_text=["我喜欢看电影。", "今天天气很好。"],
    source=["I like watching movies.", "The weather is nice today."],
    target_lang="zh",
    asr_text=["我喜欢看电影。", "今天天气很好。"],
)

print(scores)

ASR translation-quality evaluation is enabled by default and runs only when target_audio or precomputed asr_text is supplied. To transcribe generated speech directly, pass a file, a list of files, or a directory as target_audio; Whisper defaults to medium and can be changed with TranslationEvaluator(whisper_model="large-v3"). Set use_asr=False to disable every ASR_ metric.

Examples

Complete parameter templates are kept in examples/. The README intentionally stays compact; use these files for configurable parameters, input formats, and output fields.

• examples/python/translation_eval.py
• examples/python/speech_quality_eval.py
• examples/python/speaker_similarity_eval.py
• examples/python/emotion_eval.py
• examples/python/paralinguistic_eval.py
• examples/python/paralinguistic_identity_baseline.py
• examples/python/temporal_consistency_eval.py
• examples/python/latency_eval.py
• examples/bash/install_extras.sh
• examples/bash/run_latency_cli.sh

Latency can also be run from the module CLI:

python -m openstbench.latency.cli --help

Conventions

• Text inputs generally accept list[str], one-sample-per-line .txt files, and .json files where supported by the evaluator.
• Audio inputs generally accept folders, list[str], .txt path lists, and .json path lists where supported by the evaluator.
• For zh, ja, and ko, speech consistency reports CER_Consistency; other languages report WER_Consistency.
• Evaluators that accept pretrained model sources use a local-first rule. If the supplied local path exists, OpenSTBench uses it; otherwise it falls back to the configured remote model id.
• Optional dependencies are loaded only when the corresponding evaluator needs them.
• MetricX is enabled by default in TranslationEvaluator. It follows the official google-research/metricx README, uses text only, reports error scores in [0, 25] where lower is better, and can be disabled with use_metricx=False.
• TranslationEvaluator also enables ASR variants by default. With target_audio, Whisper transcribes the generated speech before the same enabled translation metrics are applied; with asr_text, the supplied transcript is used directly. Calls without either input preserve the text-only behavior.

Acknowledgements

• We especially thank SimulEval, from which parts of OpenSTBench's latency evaluation components are adapted
• sacreBLEU, COMET, MetricX, and bleurt-pytorch, a PyTorch port of BLEURT, for translation quality evaluation
• Whisper, SpeechMOS/UTMOS, Resemblyzer, and WavLM for speech quality and speaker similarity evaluation
• FunASR and Emotion2Vec for emotion preservation evaluation
• CLAP and Hugging Face Transformers for paralinguistic event evaluation

Citation

If you find our work useful, please cite as：

@misc{an2026openstbenchsemanticevaluationspeech,
      title={OpenSTBench: Beyond Semantic Evaluation for Speech Translation}, 
      author={Yanjie An and Yuxiang Zhao and Yichi Zhang and Qixi Zheng and Yujie Tu and Keqi Deng and Kai Yu and Xie Chen},
      year={2026},
      eprint={2605.30792},
      archivePrefix={arXiv},
      primaryClass={eess.AS},
      url={https://arxiv.org/abs/2605.30792}, 
}

License

OpenSTBench's original code is released under the MIT License. See LICENSE.

Some latency evaluation components include code adapted from SimulEval, which is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0). Those adapted portions are distributed under CC BY-SA 4.0. See THIRD_PARTY_NOTICES.md for details.

The datasets referenced by OpenSTBench, including the datasets used in the paper, are not covered by the OpenSTBench code license. They are provided by their original authors or distributors under their own licenses and terms of use. Some datasets are restricted to research or non-commercial use.