RVCBench

Benchmarking the Robustness of Voice Cloning

Paper · Website · Dataset · Demo · Evaluate your model · Reproduce the paper

News

• 2026-10 · pip install rvcbench: score your own model from ZipVoice- or Seed-TTS-style batch lists, compare models in one table, or use the metrics in your own code.
• 2026-09 · RVCBench is accepted to NeurIPS 2026.

Voice cloning models sound convincing in clean demos. RVCBench measures how they hold up in deployment: noisy, accented or overlapping reference audio; irregular or scam text; long-form and multilingual speech; compression; and anti-cloning protection, with and without denoising. The paper defines 18 robustness evaluations over 14,370 utterances from 204 speakers and evaluates 18 open-source models.

Evaluate your model

Generate speech with your own code, then let RVCBench score it. With ZipVoice, for example:

pip install "rvcbench[eval]"
rvcbench setup-scorers                                # once: download the metric models
rvcbench prompts --suite core-v1 --output prompts/    # reference clips and texts to synthesize
python3 -m zipvoice.bin.infer_zipvoice --model-name zipvoice \
  --test-list prompts/prompts.tsv --res-dir outputs/zipvoice
rvcbench score --suite core-v1 --generated outputs/zipvoice --output results/zipvoice --device cuda

• Works with batch scripts. prompts/ has prompts.tsv (ZipVoice), prompts.lst (Seed-TTS-eval scripts such as F5-TTS and CosyVoice) and prompts.jsonl. Save each output as <id>.wav.
• Compare models in one run. Pass several output folders to rvcbench score and get comparison.md.
• Pinned inputs. Every input is checked by hash against a fixed dataset revision, and target recordings are never exported.

The same metrics are available in your own code, with the same models and definitions:

from rvcbench import metrics

with metrics.Evaluator(["sim", "wer", "speechmos"], device="cuda") as evaluator:
    evaluator.score("generated.wav", reference="reference.wav", text="Hello there.", language="en")

See the metrics API for every metric and the one-line functions.

Requires Linux, Python 3.10+ and FFmpeg; a GPU is recommended for scoring. More in Evaluate your own model and the Core suite guide.

What it measures

Metrics: speaker similarity (SIM, ECAPA-TDNN), naturalness (MOS, UTMOS), word error rate (WER, Whisper medium), mel-cepstral distortion (MCD), emotion consistency and STOI.

Results

From the paper (arXiv v3). SIM, MOS, WER and MCD are on clean LibriTTS prompts; the last three columns are speaker similarity on other datasets. Bold marks the best value per column, and † marks models reported in the paper's appendix.

Results for compression, deepfake detectability, long-form and expressive speech are in the paper and on the website.

Reproduce the paper

To reproduce the paper, use the code released with it, kept on the v1 branch (tag v1.0). This branch, main, is v2: the same benchmark rebuilt as an installable package for evaluating new models.

git clone --branch v1 https://github.com/Nanboy-Ronan/RVCBench.git RVCBench-v1

Run the built-in models

RVCBench includes adapters for 32 voice cloning models, 5 protection methods and a denoising stage. Each model runs in its own environment from envs/.

git clone https://github.com/Nanboy-Ronan/RVCBench.git && cd RVCBench
python -m pip install -e '.[eval]'
rvcbench run --config-name ots_vc/clean/libritts/qwen3_tts_ots dataset.speaker_id=1089 adversary.max_samples=5

Every run writes a per-sample run_manifest.json with input and output hashes, seeds, failures and metric coverage. See Running the built-in models and the run guide.

Documentation

Citation

@inproceedings{jin2026rvcbench,
  title     = {RVCBench: Benchmarking the Robustness of Voice Cloning Across Modern Audio Generation Models},
  author    = {Jin, Ruinan and Liao, Xinting and Yu, Hanlin and Pandya, Deval and Li, Xiaoxiao},
  booktitle = {Advances in Neural Information Processing Systems},
  url       = {https://arxiv.org/abs/2602.00443},
  year      = {2026}
}

License

CC0-1.0. Model checkpoints, upstream code and source corpora keep their own licenses. Questions and contributions are welcome through issues and pull requests, or at ruinanjin@alumni.ubc.ca.