tts-trainer

Train your own Cantonese CustomVoice model for the ggk engine: a LoRA fine-tune of Qwen/Qwen3-TTS-12Hz-1.7B-Base on CC-0 Cantonese speech (Mozilla Common Voice yue, through its Hugging Face parquet mirrors), merged back into the base weights with the chosen speakers baked in as named voices, and exported as the GGUF that ggk runs exactly like hk-cantonese-v0.0.5 — with as many female and male voices as the data supports instead of two.

pip install tts-trainer
tts-trainer                 # opens http://127.0.0.1:8656/ in the browser

The GUI runs in your browser against a local backend, in the style of gguf-trainer. Nothing is uploaded: models, datasets and outputs are addressed by filesystem path.

What it does

The recipe follows the official Qwen3-TTS fine-tune and the hobpond/qwen3-tts-cantonese-recipe behind jmtl/hk-cantonese-cc0-qwen3-tts, generalized to several voices:

1. Select — scan the dataset parquet files (audio, sentence, client_id, gender, down_votes), keep clips of 3–20 s with colloquial Cantonese transcripts and no down-votes, and choose the voices: the contributors with the most usable clips per self-reported gender (n_female + n_male, default 3 + 3), named yue_f1 … yue_m3. Contributor ids never leave the project folder. Your own recordings can be added as a JSONL ({"audio", "text", "speaker", "gender"}); each local speaker becomes a voice under its own name.
2. Encode — decode, resample to 24 kHz, trim edge silence (speaker-set clips are padded back to length), run the 12 Hz speech tokenizer (16 codebooks) and the base model's ECAPA speaker encoder; shards go to shards/. Each voice's mean speaker embedding (fp32) is the row that gets baked.
3. Train: language LoRA — rank 32 / alpha 64 on the 196 q/k/v/o/gate/up/down projections of the talker (never the code predictor), one epoch over every usable clip, speaker slot = the clip's own embedding. Loss = codebook-0 cross-entropy with the recipe's label-shift fixes (sub-talker weight 0 by default).
4. Train: speaker LoRA — the language adapter merged in, a fresh LoRA over the voices' clips (3 epochs), speaker slot = the voice mean, i.e. exactly what the exported model is conditioned with.
5. Export — both adapters merged into the base weights, voice means written at codec_embedding rows 3000, 3001, …, tts_model_type: custom_voice with the spk_id map, speaker encoder dropped → export/<name>/ (loads in qwen_tts). Plus export/adapter/: one rank-64 adapter in PEFT layout (the two phases block-concatenated, exact), extra_tensors.safetensors with the baked rows and apply_adapter.py — the hk-cantonese release layout, so the voice ships as a 140 MB adapter if you want.
6. Convert — <name>-f16.gguf (and q8_0, q4_k) with qwen3-tts.cpp's converter, optionally the codec GGUF too.
7. Evaluate — a few Cantonese sentences per voice synthesized with the merged checkpoint (runaway / length statistics, playable in the Output tab; CER with an optional Qwen3-ASR checkpoint).

ggk -m yue_voices-f16.gguf --codec-model qwen-tokenizer-12hz-q8_0.gguf \
    --speaker yue_f1 --language chinese -t "今日天氣好好，我哋出去行下街啦。" -o out.wav

Keep --language chinese (the adapter carries the Cantonese) and split long text at punctuation, about 45 characters a chunk.

Materials

• Qwen/Qwen3-TTS-12Hz-1.7B-Base (~3.9 GB, Apache-2.0) — the Base model with the speaker encoder; CustomVoice / VoiceDesign checkpoints cannot be trained from.
• Qwen/Qwen3-TTS-Tokenizer-12Hz (~0.7 GB) — the speech tokenizer.
• Dataset parquet files, e.g. JackyHoCL/common_voice_22_yue (Common Voice yue 22.0, CC-0; ~190k validated clips in 32 files of ~1.4 GB). files/split limits how many are downloaded; 4 files are enough for a first model, more clips give more voices to choose from.

Copies already on disk are linked instead of downloaded again (the directory the GUI was started from, other projects, TTS_TRAINER_MATERIALS); downloads are detached and resumable.

Headless

tts-trainer new      --project ~/tts-trainer/projects/yue_voices
tts-trainer download --project ~/tts-trainer/projects/yue_voices
tts-trainer run      --project ~/tts-trainer/projects/yue_voices      # foreground
tts-trainer start    --project ~/tts-trainer/projects/yue_voices      # detached
tts-trainer stop     --project ~/tts-trainer/projects/yue_voices      # saves, then exits
tts-trainer status   --project ~/tts-trainer/projects/yue_voices
tts-trainer say      --project ~/tts-trainer/projects/yue_voices --speaker yue_m1 --text "唔好意思，請問附近有冇地鐵站？" --out m1.wav
tts-trainer start    --project ... --only export convert eval --force  # re-merge from the adapters
tts-trainer convert  --input export/yue_voices --output yue_voices-q8_0.gguf --type q8_0
tts-trainer convert  --input export/yue_voices --output yue_voices-q4_k.gguf --type q4_k

Every stage is idempotent and checkpointed: a reboot, a Stop or a crash resumes at the next shard / training step. project.json holds every setting; epochs of a running phase can be changed there (or in the GUI) and take effect within a log interval.

Hardware

The 1.7B talker trains in bf16 with fp32 LoRA weights: about 6 GB of VRAM at batch 4 (24 GB cards take batch 8–16), ~100 optimizer steps a minute on an RTX 5090 at batch 4. The default run (30k language clips, one epoch; 6 voices × ≤800 clips, three epochs) is roughly 7.5k + 3.5k steps. Encoding runs at ~40 clips/s on a GPU. 6 GB cards: batch 1–2 with gradient checkpointing on.

Notes

• The Qwen3-TTS model code is vendored (Apache-2.0, tts_trainer/qwen3_tts/) and patched to import under transformers 5.x as well as 4.57; no qwen-tts / peft dependency. Synthesis uses its own decode loop (synth.py), not generate().
• GGUF files are written with gguf-connector (writer, quant and reader in one package, as in gguf-trainer); the converters quantize to q8_0 themselves and the gguf package from llama.cpp is no longer needed.
• q4_k is encoded by tts_trainer/kquant.py, a numpy port of ggml's reference Q4_K quantizer that is byte-identical to quantize_row_q4_K_ref (gguf-connector up to 3.7.5 only dequantizes k-quants). The talker q4_k is a Q4_K_M-style mix: attention V / FFN down, the codec head and the small code predictor stay q8_0, the rest is Q4_K (~36 % of f16). Pure Q4_K made the fine-tuned talker run on past its sentence (2 of 8 seeds hit the 60 s cap, 3 more spoke 2-3x too long); the mix never hit the cap and stayed in the f16/q8_0 length range on 7 of 8 seeds. Expect q4_k to drift more than q8_0 — check lengths in the Output tab before you ship it. In the codec every linear/transformer weight is Q4_K and the conv kernels (row = kernel size) stay f16; rows that fit no block type are written as f16 and logged, never silently.
• Common Voice audio is used under its CC-0 release; the Mozilla Data Collective terms forbid re-hosting the clips or identifying contributors — the project keeps the mapping voice → contributor in data/voices.json only.

Production repos

• cantonese-v1-dev
• cantonese-gguf