streaming-vits

Streaming inference for VITS / piper TTS models.

Feed in a whole paragraph, get audio out incrementally — and the result is numerically identical to a full-paragraph forward pass. No sentence splitting, no prosody loss.

A drop-in replacement for sherpa-onnx-offline-tts: same model file, same flags, same WAV. It just starts producing audio before synthesis finishes.

- sherpa-onnx-offline-tts \
+ streaming-vits-offline-tts \
      --vits-model=./en_US-libritts_r-medium.onnx \
      --vits-tokens=./tokens.txt \
      --vits-data-dir=./espeak-ng-data \
      --num-threads=4 --sid=0 \
      --output-filename=./test-0.wav \
      "The quick brown fox jumped skillfully over the lazy dog..."

Elapsed seconds: 1.012
Audio duration: 9.532 s
Real-time factor (RTF): 1.012/9.532 = 0.106
Time to first audio: 141 ms          <- 9.5 s of audio, first sound in 141 ms

Add --play to hear it live as it generates.

Why this is possible

VITS is not autoregressive, so streaming looks like it shouldn't work. But SynthesizerTrn.infer() splits cleanly at the monotonic alignment:

Everything that carries paragraph-level prosody lives in the cheap frontend, which runs once on the whole input. The expensive half is a plain CNN over the frame axis — no recurrence, no global attention — so it can be evaluated in chunks, and with enough overlap context the chunks concatenate exactly.

This is the opposite trade-off from sentence chunking. Splitting text throws away the global context that produces good prosody. Splitting in frame space costs nothing, because the frame-space network cannot see beyond its receptive field anyway.

One detail makes it exact rather than approximate: both RandomNormalLike nodes (duration noise and prior noise) land in the frontend, so all sampling happens once, for every frame, before any chunk is decoded. Two overlapping chunk decodes necessarily see identical noise on the frames they share.

Install

pip install streaming-vits

# phonemization backend (one of):
brew install espeak-ng          # macOS
apt install espeak-ng           # debian/ubuntu
pip install piper-phonemize     # faster, no subprocess

pip install 'streaming-vits[play]'   # optional: live playback

Point it at any piper voice — the same tarballs sherpa-onnx uses:

wget https://github.com/k2-fsa/sherpa-onnx/releases/download/tts-models/vits-piper-en_US-libritts_r-medium.tar.bz2
tar xf vits-piper-en_US-libritts_r-medium.tar.bz2

The monolithic .onnx is split into a frontend and a decoder graph on first use and cached next to the model. You don't have to do anything.

Calibrate for your device

The chunk schedule is the whole game for latency, and the right one depends on how fast the device is. Run this once per device:

streaming-vits calibrate \
    --vits-model=./en_US-libritts_r-medium.onnx \
    --vits-tokens=./tokens.txt \
    --vits-data-dir=./espeak-ng-data \
    --num-threads=4

It measures two different things:

Margins are a property of the model's weights — how much context does the decoder actually need before its output stops changing? Swept empirically, because it is not guessable: trained weights use far more of their nominal receptive field than an untrained network suggests.

[1/3] receptive field (model property, device independent)
  seed 1234 -- deterministic across runs
  lookahead  rel RMS err
          4     32.835%
          8     11.672%
         12      4.197%
         16      1.047%
         20      0.174%
         24      0.006%
  -> left 34, right 28 (325 ms lookahead)

The frontend sampling is pinned to a fixed seed for this sweep (--seed), so repeated calibrations of the same model give the same margins. Normal synthesis is unaffected and stays stochastic.

Schedule is a property of the device. An affine cost model decode(n) = α + β·(n + margins) is fitted, then the schedule that reaches the speaker soonest without ever starving playback is solved for:

[2/3] decode cost on this device
  fit: 1.26 ms/call + 0.442 ms/frame (a frame is 11.61 ms of audio)

  predicted time to first audio  162 ms
  steady-state RTF per chunk     0.052
  survives a device              2.0x slower than this one
  steady-state decode overhead   1.21x (the ramp costs more early)

--safety N sets how much slower than measured to plan for; the prebuffer is sized so headroom is a guarantee rather than an observation. Raise it on devices with contended CPUs.

Margins are reproducible run to run, but the schedule is not quite: it depends on wall-clock timings, which move with machine load. Calibrate on an otherwise idle device, and treat --safety as the thing that absorbs the rest.

The profile is written next to the model and picked up automatically.

Results

en_US-libritts_r-medium, 4 threads, M-series Mac, 15.2 s utterance:

non-streaming  0.701s  RTF 0.046   <- you wait this long before any sound

time to first audio  0.163s   (4.3x sooner than non-streaming)
total wall clock     1.301s  RTF 0.085  (1.86x the work)
playback             no underrun

Time-to-first-audio for the non-streaming path scales with the length of the paragraph. For the streamed path it is constant, so the gap widens the more you ask it to say.

Equivalence, on the real weights:

$ streaming-vits verify --vits-model=en_US-libritts_r-medium.onnx ...
  max |streamed - full| = 8.00e-05  OK

$ streaming-vits verify --vits-model=es_MX-claude-high.onnx ...
  max |streamed - full| = 1.79e-05  OK

Commands

Accepted-and-ignored sherpa flags (--vits-lexicon, --vits-dict-dir, --tts-rule-fsts, --max-num-sentences) print a note rather than failing, so existing command lines keep working.

Python API

from streaming_vits import StreamingTTS

tts = StreamingTTS("en_US-libritts_r-medium.onnx", tokens_path="tokens.txt")

for chunk in tts.stream("Hello there. This plays before it has finished."):
    speaker.write(chunk.audio)      # float32, mono, tts.sample_rate
    print(chunk.index, chunk.frames, chunk.ready_at)

audio = tts.synthesize("Same thing, one array.")

Limits

• Extra work. Each chunk re-decodes its margin frames: ~1.2× in steady state, more during the ramp. Cached-state streaming convolutions would cut this to ~1.05× but are considerably more implementation.
• TTFA is dominated by the frontend, not by chunk size. On the numbers above the frontend is ~130 ms of a 162 ms budget; smaller first chunks will not help. Speeding up the stochastic duration predictor is the next lever.
• Streaming text in is a separate problem. The duration predictor needs the whole utterance before any audio exists. If text arrives from an LLM token stream you still chunk at clause level — but you can feed the previous clause as context and discard its audio to soften the seam.
• The bundled phonemizer is pragmatic. It shells out to espeak-ng per sentence and re-inserts punctuation, where piper drives libespeak-ng directly and gets clause terminators back. Install piper-phonemize for the faithful path. This does not affect the streaming claim — verify compares on identical phoneme ids.
• Verified on two piper voices: en_US-libritts_r-medium (904 speakers) and es_MX-claude-high (single speaker, different exporter version). Other VITS exports should work if the graph has the same seam; streaming-vits info will tell you before you rely on it.

How the split is found

streaming_vits/graph.py locates the seam structurally rather than by hardcoded node names: find the last RandomNormalLike (the prior noise), walk forward to the Add that forms z_p, then collect whatever the downstream subgraph still needs from upstream. It also reads the decoder config back off the weights — piper's config.json does not record it, and it varies by voice (libritts_r-medium uses 3 upsample stages, [8,8,4], not the 4 in the reference VITS config).

Two things are deliberately not read from node names, because those vary between piper exporter versions even when the weights do not: the upsample config comes from dec.ups.N.weight, and the seam comes from graph topology. Single-speaker voices have no speaker embedding and no sid input, so the decoder takes two inputs instead of three; that is detected rather than configured.

research/ holds the original proof of concept, including verify_streaming.py, which proves the same equivalence against the reference PyTorch VITS implementation independently of ONNX.

License

MIT