omnivoice-fastpath

Inference speedups for OmniVoice TTS, with the measurements behind them. 4.6x faster than the stock Python API on the same hardware, at no quality difference we could detect over 48 paired sentences.

Half of this repo is negative results. Batching, ONNX fp32, CUDA graphs and CFG interval all sound like they should help. None of them do, and the numbers explaining why are more useful than another list of tricks.

Numbers

Apple M2 Max, MPS, fp16, 10 voice-cloning segments of 3-4s each.

The middle row is the honest baseline: 4.6x. The top row is 8.6x, but a third of that gap is just not reloading a 813M-parameter model for every sentence, which any serving setup fixes on its own.

Where the 4.6x comes from:

Multiplied out that is 4.12x; measured is 4.61x, because torch.compile pays off more once the other levers have made each step cheaper.

Install

Not on PyPI yet — install from git:

pip install git+https://github.com/badrabbitt/omnivoice-fastpath

Or clone it, which you want anyway if you plan to run the benchmarks:

git clone https://github.com/badrabbitt/omnivoice-fastpath
cd omnivoice-fastpath && pip install -e ".[onnx,eval]"

omnivoice>=0.2.1 comes in as a dependency. Nothing here needs the OmniVoice source tree.

Use

import torch
from omnivoice.models.omnivoice import OmniVoice
from omnivoice_fastpath import apply_trimmed_cfg

model = OmniVoice.from_pretrained("k2-fsa/OmniVoice", device_map="mps", dtype=torch.float16)
apply_trimmed_cfg(model)                  # skip on CUDA, see below
model.llm = torch.compile(model.llm, dynamic=True)

prompt = model.create_voice_clone_prompt(ref_audio="ref_3s.wav", ref_text="...")
audio = model.generate(text="...", language="vi", voice_clone_prompt=prompt, num_step=12)

Reuse prompt across segments. Encoding a reference costs 1-2s, and a dubbing job is usually hundreds of segments sharing one voice.

Trimmed CFG

The one thing here that is not just parameter choice.

Every diffusion step runs a single forward over 2B rows: B conditional rows of length c_len (reference + text + target) and B unconditional rows of length u_len (target only). Both halves sit in one tensor, so the shorter unconditional rows get padded up to max_c_len. In voice-cloning mode we measured c_len=295 against u_len=89 — the unconditional half was doing 3.3x more work than it needed to, on every step.

Those padded positions attend only to themselves, and no real position attends to them, so dropping them cannot change the result. Verified at the logits level rather than argued: on CPU/fp32 the maximum absolute difference against the unpatched model is exactly 0.00000 with 100% argmax agreement. Under fp16 the difference is 0.22 with 98.4% agreement, which is GEMM reduction order changing with the tensor shape, not a different computation.

from omnivoice_fastpath import apply_trimmed_cfg
apply_trimmed_cfg(model)

It wraps forward, so upstream source stays untouched.

Turn it off on CUDA. Fewer tokens but twice the kernel launches: that trade wins where compute is the bottleneck and loses where launch latency is.

Steps

12 is the floor. 8 breaks.

Paired evaluation, 48 sentences, same seed per sentence across configs, judged by faster-whisper large-v3, 95% CI from a paired bootstrap over per-sentence WER.

12 steps is indistinguishable from 32 and 2.34x faster. Do not read 2.51% as better than 32 steps — the interval spans zero, that is noise. At 8 steps WER nearly doubles and sentences transcribed perfectly drop from 32/48 to 24/48.

Reference length

A reference clip is prepended to the sequence on every step, so its length is multiplied by the step count. Natural clips, each with its exact transcript:

Cutting 6.2s to 3.2s of the same speaker, with the transcript cut at the matching word boundary, showed no WER difference across 48 sentences (Δ −0.02, CI [−0.95, +0.92], p=0.754). WER does not measure timbre similarity, so listen before committing to a short reference for a voice you care about.

One trap: if you trim the audio and guess where to cut the transcript, the model's pacing estimate goes wrong and generates longer audio, which inflates RTF and makes the speedup look better than it is. Our first pass reported 1.86x for this reason. Cut on a word boundary and use the exact transcript.

What did not work

Batching deserves a note, since it is the first thing everyone tries. Two reasons it fails: CFG already doubles the batch, so a single item saturates the device; and _generate_iterative scores tokens in a per-item Python loop each step, so batch 32 over 32 steps means 1024 small serialized operations. Making batching pay would mean vectorizing that loop upstream.

Devices

32 short voice-cloning segments, 32 steps, batch 1.

Voice cloning is much more expensive than auto-voice mode and the bill lands almost entirely on CPU: the same sentences run at RTF 4.39 without a reference and 9.85 with one, because the reference lengthens the sequence and attention is quadratic. Any RTF you see quoted for auto-voice is optimistic for real dubbing.

Long-form is cheaper per second than short segments on every device (T4: 0.33 vs 0.45) because each internal chunk is ~8s, so fixed per-call cost amortizes. Memory stays flat with length, so hours-long narration will not OOM. Hand the whole text to one generate() call and let OmniVoice chunk it — chunking it yourself and batching the pieces is slower.

Reproducing

pip install -e ".[onnx,eval]"

# headline comparison
python benchmarks/bench_vs_upstream.py --device mps --n 10 \
  --long-ref refs/ref_full.wav --long-ref-text "..." \
  --short-ref refs/ref_3s.wav --short-ref-text "..."

# cumulative effect of each lever
python benchmarks/bench_stack.py --device mps --long-ref ... --short-ref ...

# prove trimmed CFG changes nothing, at the logits level
python benchmarks/check_fastpath_parity.py --device cpu --dtype fp32 --ref ... --ref-text "..."

Quality evaluation is three steps: generate a set per config, transcribe with faster-whisper large-v3, then compare paired.

python eval/eval_stat_gen.py torch A_step32 32
python eval/eval_stat_asr.py A_step32 B_step12
python eval/eval_stat_report.py A_step32

The corpus is 48 Vietnamese sentences from a maths-explainer domain in benchmarks/eval_sentences.py. Swap it for your own language and content; the speed results are language-independent but the WER numbers are not.

Method notes

Everything above is measured on one machine (M2 Max, 12 CPU, 64GB; Docker limited to 12 CPU / 12GB) and one Colab T4. Numbers will move on other hardware, and the CPU/GPU trade-offs behind trimmed CFG and batching could invert on a different device.

Two things worth knowing if you extend this work.

The ASR judge matters more than the model under test. An earlier version of this evaluation used faster-whisper base and reported 20-33% WER with a completely different ranking of the configurations. large-v3 puts the same audio at 2-3%. If your judge is worse than your system, you are measuring the judge.

At these WER levels most sentences are ties, so the interesting statistic is how many sentences differ at all — 12 steps differs from 32 on 12 of 48 sentences, and only those carry any signal. The minimum difference this design can detect is about 1.3 WER points, so treat anything smaller as unresolved rather than equal.

Still on the table

The prefix — reference audio plus text — is 61-64% of the sequence and is completely static across every step, yet gets recomputed each time. Caching its KV states is what dLLM-Cache, Fast-dLLM and Elastic-Cache do for diffusion LLMs, all training-free. Counting tokens puts the ceiling here around 1.8x. It is an approximation — the prefix attends to the target bidirectionally, so freezing its KV changes the result — and nobody has published this for diffusion TTS specifically.

Guidance distillation removes the unconditional forward entirely for up to 2x, but needs fine-tuning.

Credits

OmniVoice is by the k2-fsa team, Apache-2.0. This repo adds nothing to the model — it wraps the published API and measures it. Benchmarks ran against 0.2.1 (commit 38e992b).

Apache-2.0.