streaming-vits

Streaming inference for VITS / piper TTS models.

Feed in a whole paragraph, get audio out incrementally — and the result is numerically identical to a full-paragraph forward pass. No text splitting of any kind: sentences stay whole, so the intra-sentence breaks that cost VITS its intonation are never needed.

A drop-in replacement for sherpa-onnx-offline-tts: same model file, same flags, same WAV. It just starts producing audio before synthesis finishes.

- sherpa-onnx-offline-tts \
+ streaming-vits-offline-tts \
      --vits-model=./en_US-libritts_r-medium.onnx \
      --vits-tokens=./tokens.txt \
      --vits-data-dir=./espeak-ng-data \
      --num-threads=4 --sid=0 \
      --output-filename=./test-0.wav \
      "The quick brown fox jumped skillfully over the lazy dog..."

Elapsed seconds: 1.012
Audio duration: 9.532 s
Real-time factor (RTF): 1.012/9.532 = 0.106
Time to first audio: 141 ms          <- 9.5 s of audio, first sound in 141 ms

Add --play to hear it live as it generates.

Why this is possible

VITS is not autoregressive, so streaming looks like it shouldn't work. But SynthesizerTrn.infer() splits cleanly at the monotonic alignment:

half	what it does	context needed	cost
frontend	text encoder → duration predictor → alignment → prior expansion → z_p	global, over text	~15-20% of compute, and quadratic in text length
decoder	flow → HiFi-GAN	finite receptive field over frames	the rest, linear in output length

The frontend being quadratic matters: measured on en_US-libritts_r-medium, it is 16% of total compute for a 7 s utterance, 37% at 59 s, and 68% at 178 s (scaling exponent 1.89 against the decoder's 1.06). It is cheap at sentence and paragraph scale and expensive at essay scale, so very long inputs should be cut at sentence boundaries into segments — see Speaker below, which does that automatically. Chunked decoding is still exact within each segment.

Everything that carries paragraph-level prosody lives in the cheap frontend, which runs once on the whole input. The expensive half is a plain CNN over the frame axis — no recurrence, no global attention — so it can be evaluated in chunks, and with enough overlap context the chunks concatenate exactly.

This is the opposite trade-off from sentence chunking. Splitting text throws away the global context that produces good prosody. Splitting in frame space costs nothing, because the frame-space network cannot see beyond its receptive field anyway.

One detail makes it exact rather than approximate: both RandomNormalLike nodes (duration noise and prior noise) land in the frontend, so all sampling happens once, for every frame, before any chunk is decoded. Two overlapping chunk decodes necessarily see identical noise on the frames they share.

Install

pip install streaming-vits

# phonemization backend (one of):
brew install espeak-ng          # macOS
apt install espeak-ng           # debian/ubuntu
pip install piper-phonemize     # faster, no subprocess

pip install 'streaming-vits[play]'     # optional: live playback
pip install 'streaming-vits[export]'   # optional: build the cached-state decoder

Point it at any piper voice — the same tarballs sherpa-onnx uses:

wget https://github.com/k2-fsa/sherpa-onnx/releases/download/tts-models/vits-piper-en_US-libritts_r-medium.tar.bz2
tar xf vits-piper-en_US-libritts_r-medium.tar.bz2

The monolithic .onnx is split into a frontend and a decoder graph on first use and cached next to the model. You don't have to do anything.

Calibrate for your device

The chunk schedule is the whole game for latency, and the right one depends on how fast the device is. Run this once per device:

streaming-vits calibrate \
    --vits-model=./en_US-libritts_r-medium.onnx \
    --vits-tokens=./tokens.txt \
    --vits-data-dir=./espeak-ng-data \
    --num-threads=4

It measures two different things:

Margins are a property of the model's weights — how much context does the decoder actually need before its output stops changing? Swept empirically, because it is not guessable: trained weights use far more of their nominal receptive field than an untrained network suggests.

[1/3] receptive field (model property, device independent)
  seed 1234 -- deterministic across runs
  lookahead  rel RMS err
          4     32.835%
          8     11.672%
         12      4.197%
         16      1.047%
         20      0.174%
         24      0.006%
  -> left 34, right 28 (325 ms lookahead)

The frontend sampling is pinned to a fixed seed for this sweep (--seed), so repeated calibrations of the same model give the same margins. Normal synthesis is unaffected and stays stochastic.

Schedule is a property of the device. An affine cost model decode(n) = α + β·(n + margins) is fitted, then the schedule that reaches the speaker soonest without ever starving playback is solved for:

[2/3] decode cost on this device
  fit: 1.26 ms/call + 0.442 ms/frame (a frame is 11.61 ms of audio)

  for a 300-phoneme reply (~547 frames, 6.4s of audio):
  predicted time to first audio  74 ms
  steady-state RTF per chunk     0.048
  survives a device              2.0x slower than this one
  steady-state decode overhead   1.00x

  without streaming, same utterance  286 ms to first audio
  streaming gets you                  74 ms  (3.8x)

The reported figure is for --reply-ids (default 300), not the long calibration paragraph: the frontend is quadratic, so which utterance you measure changes the answer substantially. Calibration also reports the non-streaming baseline and says plainly when streaming loses or barely wins — on hardware that only just outruns playback, it sometimes does.

--safety N sets how much slower than measured to plan for; the prebuffer is sized so headroom is a guarantee rather than an observation. If the device cannot hold the factor you ask for, calibration reports the ceiling it can hold and backs off to the most headroom buyable with a prebuffer worth tolerating, rather than refusing or burying it in seconds of latency.

Margins are reproducible run to run, but the schedule is not quite: it depends on wall-clock timings, which move with machine load. Calibrate on an otherwise idle device, and treat --safety as the thing that absorbs the rest.

The profile is written next to the model and picked up automatically.

The cached-state decoder

By default each chunk re-decodes left + right margin frames. That is exact but wasted, and on a device that barely outruns playback it decides everything. A one-time export replaces it with a decoder that carries its convolution state between chunks, so no frame is computed twice:

pip install 'streaming-vits[export]'
streaming-vits export-stream --vits-model=./en_US-libritts_r-medium.onnx
streaming-vits calibrate      --vits-model=./en_US-libritts_r-medium.onnx

Exporting needs torch; the resulting graph runs on onnxruntime alone, so build it on a laptop and copy the file to the device. It is picked up automatically once it sits in the model's .streaming-vits/ directory, and the windowed decoder stays as the fallback. streaming-vits info says which is live.

Measured on a Raspberry Pi 4 (4 threads, 327-phoneme reply):

	time to first audio	vs no streaming	wasted work
no streaming	5352 ms	—	—
windowed	2964 ms	1.8x	1.41x
cached-state	1336 ms	4.0x	1.00x

Two consequences worth knowing. The decoder now emits nothing until its whole right-hand receptive field has been fed, so the first chunk is floored at that (~43 frames) — smaller is wasted effort. And its per-frame cost is ~30% higher than the windowed decoder's, which lowers the maximum safety factor the device can hold; on the Pi above that ceiling is 1.14. Re-run calibrate after exporting: the two decoders have different cost models and a profile fitted to one produces the wrong schedule for the other (the CLI warns if you forget).

Results

en_US-libritts_r-medium, 4 threads, M-series Mac, 15.2 s utterance:

non-streaming  0.701s  RTF 0.046   <- you wait this long before any sound

time to first audio  0.163s   (4.3x sooner than non-streaming)
total wall clock     1.301s  RTF 0.085  (1.86x the work)
playback             no underrun

Time-to-first-audio for the non-streaming path scales with the length of the paragraph. For the streamed path it is constant, so the gap widens the more you ask it to say.

Equivalence, on the real weights:

$ streaming-vits verify --vits-model=en_US-libritts_r-medium.onnx ...
  max |streamed - full| = 8.00e-05  OK

$ streaming-vits verify --vits-model=es_MX-claude-high.onnx ...
  max |streamed - full| = 1.79e-05  OK

Commands

command	what it does
streaming-vits-offline-tts …	drop-in for sherpa-onnx-offline-tts
streaming-vits speak … --play	synthesise, optionally play live
streaming-vits calibrate …	benchmark this device, write a profile
streaming-vits bench …	streamed vs non-streaming latency
streaming-vits verify …	check chunked output still equals a full decode
streaming-vits export-stream …	build the cached-state decoder (needs torch)
streaming-vits info …	show split graphs and active profile
streaming-vits clear-cache …	delete cached split graphs

Accepted-and-ignored sherpa flags (--vits-lexicon, --vits-dict-dir, --tts-rule-fsts, --max-num-sentences) print a note rather than failing, so existing command lines keep working.

Migrating from a sherpa-onnx streaming wrapper

If you already drive sherpa_onnx.OfflineTts with a per-sentence callback, split text at a word cap for latency, and pipe chunks to paplay --raw, then streaming_vits.Speaker is a one-class swap with the same constructor, say(), stop(), warmup() and sample_rate:

- from your_pipeline import Speaker           # sherpa-onnx based
+ from streaming_vits import Speaker

  spk = Speaker(model="en_US-libritts_r-medium.onnx",
                tokens="tokens.txt",
                data_dir="espeak-ng-data",
                num_threads=2,
                player="paplay",
-               max_words=12,      # accepted and ignored: no longer needed
                speed=1.0)
  spk.warmup()
  spk.say("...")

max_words/first_words are accepted and ignored. Word-cap splitting exists to buy latency out of a text-space chunker; here the chunking is in frame space, so you get the latency and keep whole sentences. Measured on one 7.5 s sentence, 2 threads, no text splitting either way:

sherpa-style (whole sentence, then play):  0.418 s to first audio
streaming-vits (same sentence, unsplit):   0.123 s to first audio   3.4x sooner

Text is normally not segmented. max_ids (default 1600 phonemes, roughly 35 s of speech) is a latency guard for very long input, not a quality setting; set max_ids=0 to disable it entirely. Sentences are never broken either way.

stop() cancels mid-utterance and kills the player. Speaker uses PipePlayer, which needs no PortAudio — usually the difference between working and not working on a Pi. The CLI exposes the same thing:

streaming-vits speak --player=paplay --vits-model=... "text"

Python API

from streaming_vits import StreamingTTS

tts = StreamingTTS("en_US-libritts_r-medium.onnx", tokens_path="tokens.txt")

for chunk in tts.stream("Hello there. This plays before it has finished."):
    speaker.write(chunk.audio)      # float32, mono, tts.sample_rate
    print(chunk.index, chunk.frames, chunk.ready_at)

audio = tts.synthesize("Same thing, one array.")

Limits

• Extra work, unless you export the cached-state decoder. The default path re-decodes its margin frames every chunk: ~1.2-1.4× in steady state. The cached-state decoder removes it entirely (1.00×) — see above.
• TTFA is dominated by the frontend, not by chunk size, and the gap widens once the decoder is fast: on the Pi above, 743 ms of the 1336 ms is the text encoder running before any sample can exist. Smaller chunks will not help. The encoder is quadratic and not chunkable — its output converges only as ~1/context, still 2.7e-02 off with 512 tokens of context either side, against 3e-06 for the decoder at 32 frames — so this is a floor, not an oversight.
• Streaming text in is a separate problem. The duration predictor needs the whole utterance before any audio exists. If text arrives from an LLM token stream you still chunk at clause level — but you can feed the previous clause as context and discard its audio to soften the seam.
• The bundled phonemizer is pragmatic. It shells out to espeak-ng per sentence and re-inserts punctuation, where piper drives libespeak-ng directly and gets clause terminators back. Install piper-phonemize for the faithful path. This does not affect the streaming claim — verify compares on identical phoneme ids.
• Verified on two piper voices: en_US-libritts_r-medium (904 speakers) and es_MX-claude-high (single speaker, different exporter version). Other VITS exports should work if the graph has the same seam; streaming-vits info will tell you before you rely on it.

How the split is found

streaming_vits/graph.py locates the seam structurally rather than by hardcoded node names: find the last RandomNormalLike (the prior noise), walk forward to the Add that forms z_p, then collect whatever the downstream subgraph still needs from upstream. It also reads the decoder config back off the weights — piper's config.json does not record it, and it varies by voice (libritts_r-medium uses 3 upsample stages, [8,8,4], not the 4 in the reference VITS config).

Two things are deliberately not read from node names, because those vary between piper exporter versions even when the weights do not: the upsample config comes from dec.ups.N.weight, and the seam comes from graph topology. Single-speaker voices have no speaker embedding and no sid input, so the decoder takes two inputs instead of three; that is detected rather than configured.

research/ holds the original proof of concept, including verify_streaming.py, which proves the same equivalence against the reference PyTorch VITS implementation independently of ONNX.

License

MIT