vox-tts 0.19.5

English | Русский

Project page and live demo · Quick start · Models · Vox Studio · Performance · What is verified · Changelog

vox is a speech synthesis engine written in C++17. You can use it through a C ABI (include/vox.h), the vox CLI, the vox Python library, an HTTP/WebSocket server, or the vox.js browser library. It does not depend on PyTorch, ggml or any other external library. Weights are mmapped from a single .tmod file and voices are stored as .tvoice. It runs on the CPU (x86-64 AVX2, aarch64), on Vulkan (NVIDIA / AMD / Intel / Android) and on WebGPU in the browser, with f32 / f16 / bf16 / q8 / q4 weights.

Architecture

flowchart LR
    subgraph clients["Clients"]
        CLI["vox CLI"]
        PY["Python library<br/>vox.Engine"]
        SRV["HTTP / WebSocket<br/>vox.serve"]
        ABI["C ABI · include/vox.h<br/>C, C++, Rust, Go, Node"]
    end
    subgraph engine["Engine (src/engine.cpp)"]
        RES["Resource planner<br/>device · RAM/VRAM · threads"]
        SES["Sessions<br/>duplex text ⇄ audio"]
        RNG["Philox RNG<br/>seed → audio"]
    end
    subgraph fam["Model families"]
        Q["Qwen3-TTS<br/>autoregressive"]
        F["F5-TTS<br/>flow matching"]
        K["Kokoro-82M<br/>StyleTTS2 + iSTFTNet"]
        C["CosyVoice2<br/>LM + flow + HiFT"]
        CB["Chatterbox<br/>T3 + S3Gen"]
        P["Python families<br/>vox.register_family"]
    end
    subgraph hw["Backends"]
        CPU["CPU<br/>AVX2 · NEON"]
        VK["Vulkan<br/>GLSL → SPIR-V"]
    end
    TM[(".tmod<br/>mmap weights")]
    TV[(".tvoice<br/>voice")]
    CLI & PY & SRV --> ABI --> engine
    engine --> fam
    Q & F & C & CB --> CPU & VK
    K & P --> CPU
    TM -.-> fam
    TV -.-> SES
    JS["vox.js in the browser<br/>WebGPU · JS fallback"] -.same formats and seeds.-> TM

Models

Planned: XTTS, Chatterbox Multilingual / Turbo. Each family describes itself (vox inspect, engine.capabilities), and the engine takes care of chunking, crossfades and the commit policy.

Quick start

From PyPI (vox-tts; wheels for Linux x86_64 / aarch64, Windows x64 and macOS arm64):

pip install vox-tts
python -m vox serve               # CLI of the Python package; `import vox` for the API

The Linux and Windows wheels carry the Vulkan backend. It opens the Vulkan loader at run time, so the same wheel also works on a machine without a GPU driver (CPU only); the macOS wheel is CPU only. For another platform or a build tuned to your CPU, build from source (pip does it when there is no wheel, pip install --no-binary vox-tts vox-tts forces it; it needs CMake and a C++17 compiler, and the Vulkan headers for the GPU backend):

cmake -S . -B build -DVOX_NATIVE=ON && cmake --build build -j
pip install .                     # Python module (nanobind, scikit-build-core, numpy)

Vulkan is enabled automatically if its headers are found (libvulkan-dev). The SPIR-V shaders are stored in the repository (src/vk_shaders.inc), so glslang is needed only if you change them (python tools/gen_shaders.py). On Termux, run pkg install clang cmake python python-numpy and then the same steps (the aarch64 build is checked under qemu).

# 1. Hugging Face model → .tmod (q8 is the fastest on CPU; bf16 is unquantized)
python -m vox build --model ./Qwen3-TTS-12Hz-1.7B-Base --dtype bf16 --quant q8 -o model.q8.tmod

# 2. voice from a recording, natively, without torch (24 kHz WAV is best, 5–15 s of clean speech)
vox voice create ref.wav --model model.q8.tmod --text "Текст, который звучит в записи" \
    --name my_voice --lang russian --seed 42 -o voices/my_voice.tvoice

# 3. synthesis
vox say "Привет! Это проверка." --model model.q8.tmod --voice voices/my_voice.tvoice -o out.wav
vox say --stdin --stream --model model.q8.tmod --voice voices/my_voice.tvoice | ffplay -f f32le -ar 24000 -i -

# 4. server (WAV/PCM streaming, voice creation via POST /v1/voices)
python -m vox serve --model model.q8.tmod --voices ./voices --port 8080 --device vulkan

pip install misaki                      # optional: its lexicons are embedded in the .tmod for the built-in G2P
python -m vox.build_kokoro --model ./Kokoro-82M --dtype bf16 -o kokoro.tmod     # config.json, kokoro-v1_0.pth, voices/*.pt
vox say "Hello! This is Kokoro running in vox." --model kokoro.tmod --speaker af_heart -o out.wav
vox voice speaker --model kokoro.tmod --speaker "af_heart,bm_george" -o blend.tvoice        # a blend of two voices
vox phonemize --model kokoro.tmod "It's 10 o'clock, Dr. Smith."                             # check the G2P

pip install torch pyyaml onnx onnxruntime s3tokenizer   # for the converter only
python -m vox.build_cosyvoice --model ./CosyVoice2-0.5B -o cosyvoice2.tmod      # LLM bf16, flow and HiFT f32
vox voice create ref.wav --model cosyvoice2.tmod --text "Text spoken in ref.wav" -o me.tvoice
vox say "Hello! [laugh] This is CosyVoice2 in vox." --model cosyvoice2.tmod --voice me.tvoice -o out.wav
vox say "[happy] What a nice day!" --model cosyvoice2.tmod --voice me.tvoice -o happy.wav    # emotion → instruct2
vox phonemize --model cosyvoice2.tmod "It costs 25 dollars."                                 # the pieces it will speak

pip install torch safetensors   # for the converter only
python -m vox.build_chatterbox --model ./chatterbox -o chatterbox.tmod      # T3 bf16 (--quant q8: 8-bit), S3Gen f32
vox say "Hello from Chatterbox in vox." --model chatterbox.tmod --speaker default -o out.wav
vox voice create ref.wav --model chatterbox.tmod -o me.tvoice                    # no transcript needed
vox say "[excited] We won!" --model chatterbox.tmod --voice me.tvoice -o won.wav  # emotion → exaggeration / cfg_weight
vox say "Calm down." --model chatterbox.tmod --voice me.tvoice --extra '{"exaggeration":0.3,"cfg_weight":0.5}' -o calm.wav

python -m vox.build_f5 --ckpt F5TTS_v1_Base/model_1250000.safetensors --vocab F5TTS_v1_Base/vocab.txt \
    --vocos vocos-mel-24khz/ -o f5.tmod                  # DiT in f16 (since 0.18); --quant q8 [--keep-f32 adaln] for 8 bits
vox voice create ref.wav --model f5.tmod --text "Текст, который сказан в ref.wav" -o me.tvoice
vox say "Текст для озвучки" --model f5.tmod --voice me.tvoice --device vulkan -o out.wav [--nfe 32 --cfg 2 --sway -1 --speed 1]

Python:

import vox
eng = vox.Engine("model.q8.tmod", device="vulkan:0")
voice = eng.clone_voice("ref.wav", "Текст референса", name="my_voice", seed=42)
voice.save("my_voice.tvoice")
audio = eng.synthesize("Привет!", voice)                 # numpy float32, 24 kHz
with eng.session(voice, seed=42) as s:                    # text in pieces → audio as it becomes ready
    s.feed("Первая фраза. "); chunks = list(s.ready())
    s.end(); chunks += list(s)

k = vox.Engine("kokoro.tmod")
audio = k.synthesize("Hello there!", k.speaker_voice("af_heart"), speed=1.1)
k.phonemize("Hello there!")        # {'phonemes': …, 'chunks': […], 'g2p': 'misaki' | 'builtin'}

Performance

Measured by the user on vox 0.14.0: tools/run_full_check.py on a 16-thread CPU with an RTX 3050 6 GB, Qwen3-TTS 1.7B-Base, about 13 s of speech. qwentts.cpp (Q8_0) ran on the same machine with the same text. WER and CER were 0 for all nine vox variants and both qwentts.cpp runs.

F5-TTS on real weights (hotstone228/F5-TTS-Russian), a 2.3 s phrase. Since 0.13 the whole DiT step and Vocos run on the GPU:

Kokoro-82M on real weights, on the CPU:

0.14.1 speeds up every CPU matrix product with many input rows. Each output is computed with the same summation order as before, so the audio is bit for bit the same. On the 2-core development box this took Kokoro from RTF 0.89 to 0.56, and F5 from 0.52 to 0.35 (f32) or 0.42 (bf16). Your numbers will come from a new run.

CosyVoice2-0.5B (0.16, CPU only) was timed with the real dimensions and random weights (tools/make_tiny_cosyvoice.py --full) on the same 2-core box. The LLM takes 96 ms per speech token in bf16 and 59 ms with --quant q8 (25 tokens make a second of speech, so the LLM alone runs at RTF 2.4 / 1.5). The flow is the heavy part: its estimator has 56 transformer blocks and 14 residual blocks, run 10 steps × 2 (classifier-free guidance) over the prompt and the new frames together — 16 s for 2.4 s of speech with a 2.3 s prompt (≈ 1 TFLOP per 4 s of speech). HiFT adds ~0.6 s per second of audio. In total RTF ≈ 7–10 on 2 cores; a desktop CPU with 8–16 cores should be several times faster, but this model really wants a GPU, which vox does not run it on yet. These numbers say nothing about quality: the weights were random.

Chatterbox (0.17, CPU only), timed the same way (tools/make_tiny_chatterbox.py --full, 2 cores, 60 speech tokens = 2.4 s, built-in 3.2 s prompt): T3 (a 30-layer Llama, two passes per token for classifier-free guidance, run as one batch) took 15.1 s in bf16 and 9.3 s with --quant q8, prefill included — about 250 / 150 ms per token; S3Gen's flow (the same networks as CosyVoice2's) 23.7 s, HiFT 1.8 s. In total RTF ≈ 15–17 on 2 cores, peak RSS 1.8 GB (1.4 GB with q8).

On the GPU (0.18): for CosyVoice2 and Chatterbox the flow estimator — the conditional and the unconditional pass of one Euler step — is one Vulkan submission (gpu_f5.cpp, the F5 kernels plus a causal convolution and Mish), and the token LM (CosyVoice2's Qwen2, Chatterbox's T3 with both CFG passes) runs on the GPU transformer of Qwen3-TTS with its KV cache in video memory. The flow encoder and HiFT stay on the CPU (they are run once per piece). Checked on lavapipe against the CPU and the official code (the same tolerances as on the CPU); speed on a real card not measured yet. --device cpu keeps everything on the CPU; vox bench stats show flow_device, llm_device / t3_device and the GEMM in use.

How each model runs

flowchart LR
    T["text"] --> TOK["Qwen2 BPE<br/>tokenizer"]
    V[".tvoice<br/>speaker emb + ref codes"] --> PF
    TOK --> PF["prefill<br/>ICL / x-vector / speaker / instruct"]
    PF --> TK["talker<br/>28 layers, 1 token / frame"]
    TK -->|"code 0"| CP["code predictor<br/>15 passes, 1 dispatch"]
    CP -->|"16 codes / frame"| DEC["codec decoder<br/>streaming, ring KV cache"]
    DEC -->|"4 frames = 320 ms"| PCM["PCM 24 kHz"]
    TK -.next frame.-> TK

flowchart LR
    T["text"] --> SPLIT["split like infer_process<br/>(max_chars)"]
    R["reference mel<br/>from .tvoice"] --> DIT
    SPLIT --> DIT["DiT × NFE steps<br/>2 CFG branches, sway sampling"]
    N["Philox noise<br/>(seed, piece)"] --> DIT
    DIT --> VOC["Vocos<br/>→ iSTFT"]
    VOC --> XF["0.15 s crossfade<br/>between pieces"] --> PCM["PCM 24 kHz"]

flowchart LR
    T["text"] --> G2P["G2P<br/>built-in misaki port<br/>or Python misaki"]
    G2P --> CH["pieces ≤ 510 phonemes<br/>cut at . ! ? → : ; → , —"]
    VO["voice pack<br/>style = pack[len − 1]"] --> PR
    CH --> BERT["PL-BERT<br/>ALBERT, shared layer"]
    BERT --> PR["prosody predictor<br/>durations · F0 · energy"]
    CH --> TE["text encoder<br/>CNN + BiLSTM"]
    PR --> AL["alignment<br/>repeat by duration"]
    TE --> AL
    AL --> DEC["decoder<br/>AdaIN res-blocks"]
    PR -->|"F0, N"| DEC
    DEC --> GEN["iSTFTNet generator<br/>NSF sine source, Snake"]
    GEN --> PCM["PCM 24 kHz<br/>600 samples / frame"]

flowchart LR
    T["text"] --> TN["text_normalize<br/>numbers in words, groups of 60–80 tokens"]
    TN --> TOK["Qwen2 BPE"]
    V[".tvoice<br/>speech tokens · mel · x-vector"] --> LM
    TOK --> LM["Qwen2 LM 0.5B<br/>25 speech tokens / s, RAS sampling"]
    LM --> ENC["flow encoder<br/>Conformer, ×2 → 50 Hz"]
    V --> CFM
    ENC --> CFM["flow matching<br/>U-Net estimator, 10 Euler steps, CFG 0.7"]
    CFM --> HIFT["HiFT vocoder<br/>F0 → NSF source, iSTFT"]
    HIFT --> PCM["PCM 24 kHz"]

flowchart LR
    T["text"] --> PN["punc_norm<br/>tokenizer.json (char BPE)"]
    V[".tvoice<br/>speaker embedding · 6 s of speech tokens<br/>mel · tokens · x-vector"] --> COND
    COND["conditioning<br/>speaker · perceiver · exaggeration"] --> T3
    PN --> T3["T3: Llama 0.5B, 30 layers<br/>CFG, min-p, repetition penalty"]
    T3 --> ENC["flow encoder<br/>Conformer, ×2 → 50 Hz"]
    V --> CFM
    ENC --> CFM["flow matching<br/>10 Euler steps, CFG 0.7"]
    CFM --> HIFT["HiFT vocoder"]
    HIFT --> PCM["PCM 24 kHz"]

Voice profiles: several voices, emotions, any model

New in 0.15. A .tvoice is now a profile: one file holds several voices. Each voice has styles (emotions, delivery), the recordings it was cloned from, the built-in speakers to use on models that cannot clone, and a cache of the voice already built for each model it was used with.

flowchart LR
    subgraph P["cast.tvoice"]
        A["anna<br/>styles: default · happy · whisper<br/>recordings + text"]
        B["boris<br/>speakers: kokoro=am_adam<br/>description for VoiceDesign"]
        C[("cache<br/>anna/default on qwen3 base<br/>anna/happy on f5 …")]
    end
    P -->|"cached, or cloned from the recording"| Q["Qwen3 Base · F5"]
    P -->|"the speaker named for the model"| K["Kokoro · Qwen3 CustomVoice"]
    P -->|"description + style instruction"| D["Qwen3 VoiceDesign"]

vox profile new cast.tvoice --name "Cast"
vox profile add cast.tvoice --voice anna anna.wav --text "What Anna says in the recording"
vox profile add cast.tvoice --voice anna --style happy anna_happy.wav --text "Something cheerful"
vox profile set cast.tvoice --voice anna --style whisper --instruct "Whisper softly" --speed 0.95
vox profile add cast.tvoice --voice boris --from boris.tvoice          # a voice from another file (any model)
vox profile set cast.tvoice --voice boris --speaker "kokoro=am_adam;qwen3-tts/custom_voice=ryan"
vox profile show cast.tvoice
vox profile prepare cast.tvoice --model f5.tmod                        # build every voice for this model now and keep it in the file
vox say "anna: Hello! [happy] Great news.
boris: [whisper] Really? [pause 400ms] Tell me." --model f5.tmod --voice cast.tvoice -o dialogue.wav

p = vox.Profile("Cast")
p.add_recording("anna", "anna.wav", "What Anna says in the recording")
p.add_recording("anna", "anna_happy.wav", "Something cheerful", style="happy")
p.set_style("anna", "whisper", instruct="Whisper softly", speed=0.95)
p.set_voice("boris", speakers={"kokoro": "am_adam"})
p.save("cast.tvoice")
audio = engine.synthesize("Hello there.", p, voice_id="anna", style="happy")
audio = engine.synthesize("anna: Hi! [pause 300ms]\nboris: [laugh] Hello.", p)     # markup

One profile on any model. When a model needs a voice the profile does not yet have a build for, the engine builds it on first use and save() / vox profile prepare keeps it:

A voice that cannot be made on a model (for example, no Kokoro speaker named) is a clear error that says what to add.

Markup in the text (markup: auto by default, strict, off):

In auto, a bracket that is none of these stays text. In strict it is dropped, and with off everything is read as text. The text around tags is spoken exactly as if each piece were said alone. What the engine could not do as asked is listed in the session's notes (session.info["notes"]). Tags may arrive split across feed() calls.

Compatibility. A 1.0 .tvoice opens as a profile with one voice and speaks bit for bit as before. A profile whose default voice has a build is saved as format 1.1: its top level is that voice in the 1.0 layout, so vox 0.14 and vox.js read it as an ordinary voice. A profile with no build yet is format 2.0, which older versions refuse with a clear message. Recordings are stored in it once, as WAV.

Vox Studio: the engine from a browser

New in 0.15. vox web (or vox -web) starts the engine with a built-in server and prints links with a secret. Open the Vox Studio page and the engine is controlled from the browser: a script with several voices and markup, playback while it is being synthesized, voice profiles (record from the microphone, styles, built-in speakers per model), models and devices, and the history of everything spoken. The page is static (it lives at vox.arkonova.network/studio/ and is also served by the engine itself); the speech is made on your machine. The page talks to the engine with HTTP requests, the way a Clash dashboard talks to its core. 0.19: a new interface — a side rail, takes with large voice-coloured waveforms ("Another take" with a new seed, "To the editor"), who speaks as voice badges above the editor, an emotion menu, settings grouped with short explanations (a sheet on narrow screens), model cards with "Fit to this computer" (vox tune), history search, light and dark themes; profiles with the same name are told apart by their file.

vox web --models ./models --voices ./voices
  Vox Studio:  https://vox.arkonova.network/studio/#url=http%3A%2F%2F127.0.0.1%3A7860&secret=…
  this engine: http://127.0.0.1:7860/studio/#secret=…
vox web --models ./models --host 0.0.0.0 --tls      # for other computers: https with a self-signed certificate

sequenceDiagram
    participant B as Browser: Vox Studio page
    participant E as vox web (your computer)
    B->>E: GET /health
    B->>E: GET /v1/info  (Authorization: Bearer secret)
    E-->>B: models, profiles, devices
    B->>E: POST /v1/speak {text, model, profile, voice_id, params}
    E-->>B: PCM chunks as they are synthesized (played at once)
    E-->>B: X-Vox-Clip → /v1/history/<id> (notes, RTF)

• Secret. Every /v1 request needs it (Authorization: Bearer …, or ?token= for audio links and WebSockets). The secret travels in the link's #fragment, which browsers never send to the site hosting the page. It is stored in ~/.vox/web_secret; --new-secret makes a new one.
• Reaching the engine. An https page may call http://127.0.0.1 / localhost (Chrome may ask once to allow local network access). From another computer it may not call plain http, so there are three ways:
  • --tls: a self-signed certificate is made for this machine's names and addresses; open the engine's https://…/health once and accept it.
  • --cert / --key with your own certificate.
  • The Studio that the engine serves itself (http://HOST:PORT/studio/).
• API (for your own clients too): models (/v1/models, load / unload / add), profiles (/v1/profiles: create, PATCH, recordings, prepare, download / upload), POST /v1/speak (streamed pcm16 / f32 / wav), /v1/history, and WebSocket /v1/duplex (the protocol of vox serve). The full list is at the top of python/vox/web.py.

Self-tuning: vox tune

New in 0.19. Which device and placement are fastest depends on the model and the computer (an RTX 5050 runs F5 faster with one GEMM, an RTX 3050 with the other; a small model may be faster on 16 CPU cores than on a laptop GPU). vox tune measures instead of guessing:

vox tune --model cosyvoice2.tmod --voice me.tvoice       # a model with built-in speakers needs no voice
  [1/7] vulkan:0 (NVIDIA GeForce RTX 5050)        RTF 0.53, first sound 5393 ms, GEMM convgemm
  [2/7] vulkan:1 (NVIDIA GeForce RTX 3050)        RTF 0.59, first sound 5995 ms, GEMM convgemm
  [3/7] cpu, 8 threads                            stopped: slower than the best so far
  …
best: vulkan:0 (…); saved to ~/.cache/vox/tune.json
vox say "…" --model cosyvoice2.tmod --voice me.tvoice    # device auto now opens it on vulkan:0

• Candidates (vox tune --show lists them): every non-software GPU with the default placement, and with each light GPU-capable stage (under a quarter of the GPU weights: Qwen3's codec decoder, F5's Vocos) moved to the CPU; then the CPU with a thread per physical core, with every logical CPU (SMT) and with ¾ of the cores. Heavy stages on the CPU were 3–10× slower in every measurement, so they are no longer tried (--device vulkan:0,flow=cpu still works by hand). GPUs go first: a candidate whose synthesis passes twice the best one's (+3 s; the whole run is also capped at 3× the best's wall time, at least 90 s) is stopped, and once one CPU thread count is stopped the other CPU ones are not run. Results within 3 % of the fastest count as a tie: the first of them (the plain configuration) is kept. --candidates JSON measures a list of your own.
• GEMM is not a separate candidate: for F5 and the flow models every GPU candidate times convgemm against gemmb on the model's own layers when its graph is built, and keeps the faster one; the row shows which one won (GEMM gemmb) and the result records it.
• Each candidate runs in its own child process (vox bench; one Vulkan device per process), on a short text in the model's language. --goal latency keeps the earliest first sound instead of the lowest RTF; --runs N, --text.
• Applying it: an engine opened with the default resources (device auto, no threads, placement or layers) looks up the model and the hardware (CPU, cores, RAM, every GPU and its driver) and opens with the measured configuration; vox plan shows tuned: …. Anything explicit wins, and VOX_TUNE=0 or {"tuned": false} turns it off. A new GPU, a driver update or a rebuilt model are new keys.
• Python: vox.tune(model, voice) (progress callback, the same cache), vox.tune_info(model); the Studio has "Fit to this computer" on every model (POST /v1/models/tune). vox tune --forget removes a result.

Resources: device, memory, mode

vox devices [--json]                          # CPU, RAM, GPUs with indices and free video memory
vox plan --model m.tmod --mode memory --ram-limit 3GiB --vram-limit 2GiB     # plan without loading weights
vox say "…" --model m.tmod --voice v.tvoice --device gpu:1 --mode balanced --threads 6 --affinity 0-5 \
    --priority low --ram-limit 6GiB --vram-limit 3GiB --place codec=cpu --gpu-layers talker=14 --max-sessions 2
--config resources.json                       # the same as a JSON file; command-line flags override it

flowchart LR
    M["model manifest<br/>stages: weight, traffic / frame"] --> PL{{"planner<br/>src/resources.cpp"}}
    S["system<br/>cores, cgroup RAM, free VRAM"] --> PL
    U["requested<br/>device, limits, mode"] --> PL
    PL --> P1["stage → GPU<br/>(best benefit per byte first)"]
    PL --> P2["talker split by layers<br/>first k on GPU"]
    PL --> P3["threads, priority,<br/>RAM / VRAM estimate"]
    PL -->|"over a hard limit"| E["OutOfMemory<br/>with the plan"]

Duplex streaming

Text can be fed on the fly while audio is pulled as it becomes ready. It also works the other way round: type text and get back a single recording. The engine handles the plumbing: it knows how each model accepts text and decides what to hand it and when.

sequenceDiagram
    participant LLM as LLM / user typing
    participant S as vox session
    participant M as model
    LLM->>S: write("Hello, ")
    S->>M: words (Qwen3: token-level)
    M-->>S: frames
    S-->>LLM: audio chunk (320 ms)
    LLM->>S: set_text("Hello, how are you?")
    Note over S: edits to the unspoken tail are applied,<br/>spoken text is never re-spoken
    LLM->>S: end()
    S->>M: last piece (F5 / Kokoro: whole sentence)
    M-->>S: audio
    S-->>LLM: final chunks

with engine.duplex(voice, commit="auto", idle_ms=600) as d:       # Python
    d.write("Привет, "); d.write("как дела?"); d.end()           # from any thread
    for chunk in d.audio(): play(chunk)                           # aaudio() for asyncio, final() gives one recording
printf 'Привет. Как дела?' | vox live --model m.tmod --voice v.tvoice -o out.wav --commit sentence
# HTTP: WebSocket /v1/duplex — {"type":"start"|"text"|"set_text"|"flush"|"end"|"cancel"}, audio as binary PCM frames

Commit policy (commit): auto, sentence or manual (nothing goes out until flush() / end()). On a conflict with text that has already been spoken: keep | raise | respeak.

Browser: vox.js

A single script with no dependencies (docs/vox.js, ES module docs/vox.mjs) that runs F5-TTS and Qwen3-TTS on WebGPU. Without WebGPU it falls back to a JS CPU backend.

<script src="vox.js"></script>
<script type="module">
  const f5 = await Vox.load("hf:SWivid/F5-TTS", { dtype: "bf16", onProgress: (p) => console.log(p) });
  const voice = await f5.cloneVoice(refFile, "Текст, который сказан в записи");
  const { pcm, sampleRate } = await f5.synthesize("Привет, мир!", { voice, nfe_step: 16, seed: 1 });
  Vox.play(pcm, sampleRate);

  const q = await Vox.load("hf:Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice", { dtype: "bf16" });
  const v = q.speakerVoice(q.speakers[0], "", { language: "english" });
  for await (const piece of q.stream("Long text.", { voice: v, seed: 1 })) { /* audio pieces */ }
</script>

What is verified

Every family is built tiny with the official PyTorch classes and random weights (the same modules, tensor names and dataflow as the real checkpoints, only narrow), then compared with vox stage by stage. CI runs this on the CPU and on software Vulkan (lavapipe).

vox verify runs the reference codes with teacher forcing and compares every step's logits and the decoder audio. For q8 it checks mean KL (≤ 0.02) and top-5 (≥ 97%) instead of argmax.

python tools/run_full_check.py --model-dir ./Qwen3-TTS-12Hz-1.7B-Base --ref-audio ref.wav \
    --ref-text "Текст референса" --lang russian --dtypes bf16,q8 \
    --devices "cpu;vulkan:1,strict;vulkan:1,codec=cpu,strict" --threads 8 \
    --external "qwentts.cpp=./qwentts-cli --model m.gguf --out {out} < {text_file}" \
    --external-time 'qwentts.cpp=\[Perf\] Total:\s*([\d.]+)\s*ms' \
    --wer --asr-cmd 'whisper-cli -m ggml-small.bin -l {lang} -nt -np -f {wav}' --out work/

Formats

• .tmod: a VOXM header, a JSON manifest (stages, precision, tokenizer, generation parameters, voices) and blobs aligned to 4096 bytes for mmap.
• .tvoice: safetensors. Since 0.15 it is a profile (metadata profile: voices, styles, recordings, cache index; tensors ref.<id> and c<N>.*); the default voice's build sits at the top level in the 1.0 layout (speaker_emb, ref_codes, ref_text_tokens, ref_audio, metadata voice). A build is bound to the weights by sha256. A speaker voice (built-in speaker, description, Kokoro voice or blend) stores the speaker name and is tied to the model variant.
• Q4: 4 bits per weight in blocks of 32. --quant q4 covers the talker attention and MLP, q4mlp only the MLP, q4full everything. On the RTX 3050 it is faster and uses a third less VRAM, but it does not pass the KL/top-5 tolerance. WER on five phrases was 0.12–0.18 against 0.07 for q8, with no audible difference on clones.

From other languages

All bindings go through the C ABI. examples/ has C (minimal.c), C++ (minimal.cpp), Rust (rust/), Go (go/, cgo), Node.js (node/, koffi) and Python ctypes. tests/test_examples.py builds each one and requires the audio to match vox say --seed 42 bit for bit. A model can also be written entirely in Python and registered with vox.register_family: see docs/python-backends.md.

Project page

site/ holds the sources of the project page, which has a live WebGPU demo. Build it into docs/ with python tools/build_site.py. The code examples on the page come straight from examples/, so they are the same files the tests check. The charts are generated from measured numbers by python tools/make_charts.py.

Code layout

Tests

tests/run_all.sh          # C++, tokenizer, Qwen3 / F5 / Kokoro against the official code (if torch and the packages are installed),
                          # examples in 6 languages, vox.js in Chromium (if playwright is available), Python, server, duplex, Q4
VOX_QWEN2_GGUF=qwen2.gguf VOX_TEST_DEVICES="cpu;vulkan:0" python tests/test_official.py build/vox
python tests/test_kokoro_tiny.py build/vox
VOX_COSYVOICE_SRC=CosyVoice VOX_QWEN2_GGUF=qwen2.gguf python tests/test_cosyvoice_tiny.py build/vox   # a clone of FunAudioLLM/CosyVoice
python tests/test_chatterbox_tiny.py build/vox                                                        # pip install chatterbox-tts

Test vocabulary: curl -L -o qwen2.gguf https://raw.githubusercontent.com/ggml-org/llama.cpp/master/models/ggml-vocab-qwen2.gguf. CI (.github/workflows/ci.yml) runs all of this on the CPU and on lavapipe.

Limitations

• Native GPU support is Vulkan only (no CUDA or Metal). The shaders are generic, with no tensor cores or cooperative matrix. One Vulkan device per process.
• Kokoro: CPU only (no Vulkan and no browser yet). The built-in G2P is English only and differs from misaki on a few words (a heuristic tagger instead of spaCy, no espeak). On real weights it is about 1.7× slower than PyTorch on 16 threads (measured on 0.14.0, before the 0.14.1 matrix-product speedup).
• F5-TTS at 32 steps is still slower than real time on the GPU (RTF 5 on an RTX 5050). Its graph needs dim_head ≤ 64, which every official checkpoint satisfies.
• CosyVoice2: on the GPU only the LLM and the flow estimator (the encoder, HiFT and the cloning frontend run on the CPU). The text frontend has no ttsfrd / wetext (Chinese numbers stay digits, as in the official code without them). Audio comes out per group of sentences, not token by token (the official bidirectional streaming mode is not implemented). Checked against the official code on a tiny model only.
• Chatterbox: on the GPU only T3 and the flow estimator (the encoder, HiFT, VoiceEncoder on the CPU); English only (not the Multilingual or Turbo models). Audio comes per group of sentences. For T3's speech prompt a cloned voice is resampled to 16 kHz with torchaudio's algorithm, the official code uses soxr: S3Gen's prompt and the speaker embedding come out the same, part of T3's 150 prompt tokens can differ. Checked against the official code on a tiny model only.
• The main path is built for autoregressive codec models. Non-AR models use the "direct" mode of Backend, as F5, Kokoro, CosyVoice2 and Chatterbox do. XTTS is not supported yet.
• The official non_streaming_mode=True (the whole text in the prefill) is not implemented. A Python backend runs on the CPU in the interpreter and is meant for prototypes.
• GPU sampling uses float and CPU sampling uses double. The same seed on the same device gives the same audio.
• Voice input resampling is sinc/Kaiser. It matches qwen_tts (soxr) bit for bit only for 24 kHz WAV.
• NFC normalization is done in the Python wrapper. From the CLI and the C ABI, pass text that is already in NFC.
• The aarch64 build is checked under qemu. The macOS and Windows RAM/core probes have not been run.

License

Apache-2.0 (see LICENSE). third_party/json.hpp is nlohmann/json (MIT). Model weights are distributed under their own licenses and are not part of the repository.