RVCBench

Comprehensive Voice Cloning Evaluation

Documentation · Paper · Website · Dataset · Demo · Evaluate your model · Reproduce the paper

News

• 2026-10 · pip install "rvcbench[eval]": score your own model from ZipVoice- or Seed-TTS-style batch lists, compare models in one table, or use the metrics in your own code.
• 2026-09 · RVCBench is accepted to NeurIPS 2026.

RVCBench is a general-purpose package for evaluating voice cloning. It brings speaker similarity, speech quality, intelligibility, pronunciation accuracy and emotion consistency into one scoring API, and provides ready-to-use datasets for evaluating a model across languages, speakers and recording conditions.

Both workflows are included in the pip package. Your model can run in its own environment or through an API; RVCBench scores the audio it produces.

First time here? Follow the complete getting started guide. It walks through installation, required files, both workflows and reading the results.

Install

python -m pip install "rvcbench[eval]"
# Linux: install FFmpeg if it is not already available (e.g. sudo apt-get install ffmpeg).
rvcbench setup-scorers

Python 3.10+ and Linux are supported. rvcbench[eval] installs the package with all seven public metrics; plain rvcbench installs the runner and datasets without the optional scoring dependencies. setup-scorers downloads and verifies the metric models once. They are reused from your local cache. A GPU is optional; choose device="cpu" or --device cpu to score on CPU.

For a smaller initial download, select only the metrics you need: rvcbench setup-scorers --metrics sim wer speechmos. See the installation guide for CPU/GPU installation, caches and troubleshooting.

Score your own audio

from rvcbench import metrics

with metrics.Evaluator(["sim", "wer", "speechmos"], device="cpu") as evaluator:
    scores = evaluator.score(
        "generated.wav",
        reference="speaker_reference.wav",  # a recording of the intended speaker
        text="Hello there.",                 # what the generated audio should say
        language="en",
    )
    print(scores)  # dictionary with sim, wer and speechmos

Reuse the evaluator for a whole dataset: call score() for each file, and each metric model loads once. For every available metric, use metrics.Evaluator("all") and also pass target="same_text_recording.wav" for MCD and STOI. The target is a recording of the same text as the synthesized audio; the speaker reference may contain different words.

The metrics guide includes one-line functions, scoring all metrics, a batch example and metric definitions. No RVCBench dataset or model adapter is needed for this workflow.

Evaluate your model

RVCBench downloads the selected data, prepares reference clips and texts, and scores the audio your model generates. To run only selected scenarios, use rvcbench tasks --suite core-v1 to list them, then pass --tasks chinese (or several task IDs) to both prompts and score. See the scenario selection guide.

Start with the 52-utterance onboarding suite:

rvcbench prompts --suite onboarding-v1 --output prompts/
# Generate every prompt with your model. Save outputs as <id>.wav in outputs/my-model/.
rvcbench score --suite onboarding-v1 --generated outputs/my-model --output results/my-model --device cpu

prompts/ contains prompts.jsonl, prompts.tsv (ZipVoice) and prompts.lst (Seed-TTS-style scripts such as F5-TTS and CosyVoice). For example, a model environment with ZipVoice installed can run:

python3 -m zipvoice.bin.infer_zipvoice --model-name zipvoice \
  --test-list prompts/prompts.tsv --res-dir outputs/my-model

The model evaluation guide shows the exact file format and an example loop for your own inference function. Model inference is the step supplied by you; data preparation, metric scoring and report generation are automatic.

Use the same suite name for prompts and score. Every input is checked against a fixed dataset revision; target recordings stay in the scoring data and are not exported with the prompts. submission.json contains per-task scores, coverage and failures. Task directories include per-sample scores and bootstrap confidence intervals. These suites are currently previews; published paper results use a separate protocol.

Resume an interrupted or partial evaluation with the original command plus --resume:

rvcbench score --suite onboarding-v1 --generated outputs/my-model --output results/my-model --device cpu --resume

Compare several models in one call, or compare reports you already have:

rvcbench score --suite core-v1 --generated outputs/model-a outputs/model-b --output results/comparison --device cuda
rvcbench compare results/model-a results/model-b --output results/compare

Comparison checks both the suite and scoring fingerprints. Different scoring environments must be rescored together, or explicitly inspected with --allow-incompatible, which disables ranking. The suite guide lists tasks, coverage and approximate scoring costs.

Evaluation coverage

The package covers clean voice cloning, multilingual and cross-lingual speech, speaker demographics, long-form speech, unusual text, background noise, overlapping speakers, compression, and protected references. Its seven public metrics cover identity, content, quality, intelligibility and emotion. The paper additionally studies deepfake detectability and an audio-LLM expression judge; these two components are not yet available through the packaged suites.

The paper defines 18 evaluations over 14,370 utterances from 204 speakers. The package also includes adapters for running existing models and tools for protection and denoising research.

Results

From the paper (arXiv v3). SIM, MOS, WER and MCD are on clean LibriTTS prompts; the last three columns are speaker similarity on other datasets. Bold marks the best value per column, and † marks models reported in the paper's appendix.

Results for compression, deepfake detectability, long-form and expressive speech are in the paper and on the website.

Reproduce the paper

To reproduce the paper, use the code released with it, kept on the v1 branch (tag v1.0). This branch, main, is v2: the same benchmark rebuilt as an installable package for evaluating new models.

git clone --branch v1 https://github.com/Nanboy-Ronan/RVCBench.git RVCBench-v1

Run the built-in models

RVCBench includes adapters for 32 voice cloning models, 5 protection methods and a denoising stage. Each model runs in its own environment from envs/.

git clone https://github.com/Nanboy-Ronan/RVCBench.git && cd RVCBench
python -m pip install -e '.[eval]'
rvcbench run --config-name ots_vc/clean/libritts/qwen3_tts_ots dataset.speaker_id=1089 adversary.max_samples=5

Every run writes a per-sample run_manifest.json with input and output hashes, seeds, failures and metric coverage. See Running the built-in models and the run guide.

Documentation

Citation

@inproceedings{jin2026rvcbench,
  title     = {RVCBench: Benchmarking the Robustness of Voice Cloning Across Modern Audio Generation Models},
  author    = {Jin, Ruinan and Liao, Xinting and Yu, Hanlin and Pandya, Deval and Li, Xiaoxiao},
  booktitle = {Advances in Neural Information Processing Systems},
  url       = {https://arxiv.org/abs/2602.00443},
  year      = {2026}
}

License

CC0-1.0. Model checkpoints, upstream code and source corpora keep their own licenses. Questions and contributions are welcome through issues and pull requests, or at ruinanjin@alumni.ubc.ca.