index-tts-2.5-mlx

IndexTTS-2.5 voice cloning on Apple Silicon, rebuilt from PyTorch onto MLX with an int8-quantized GPT decoder. Torch-free, runs entirely on the unified-memory GPU, and ships as a one-click uvx package that auto-downloads the weights from Hugging Face.

• Voice cloning from a short reference clip (≤15 s) — Chinese / English / Japanese / Cantonese, including mixed-language text with numbers and abbreviations.
• Faster than real-time (RTF ≈ 0.45) and ~2.4× faster than the official PyTorch MPS backend.
• Torch-free: numpy + MLX only. No PyTorch, no ONNX Runtime, no MNN.
• One self-contained wheel: the inference core and all MLX model ports are vendored in — pip install gives you everything except the weights.

Hardware: Apple Silicon Mac (M1 or newer), macOS 13+, Python 3.10+. MLX uses the unified-memory GPU.

---

Features

• Zero-shot voice cloning — supply any ≤15 s clean reference; the timbre and speaking style are carried into the output. Build the speaker once and reuse it across unlimited lines.
• Multilingual + code-switching — zh, en, ja, yue, and mixed text in a single sentence (e.g. Use the CPU or GPU, 都可以).
• Text normalization — numbers, abbreviations and symbols are read out correctly via wetext (e.g. 2025 年 → “二零二五年”, 100 万 → “一百万”). Disable with --no-normalization.
• Rich decoding controls — greedy or sampling (top_k / top_p / temperature / seed), repetition_penalty, duration_factor (speech rate), and the flow-matching solver knobs (n_timesteps, cfg_rate).
• Auto-download — weights pull from Hugging Face on first run and are cached for reuse; CLI and Python API share the same cache.
• Timing report — every synth prints load / clone / synth time, RTF, and a per-stage breakdown.
• Inline special tokens (experimental) — see below.

Inline tags / emotion

There are no [laugh] / [sigh] / [猪叫]-style paralinguistic tags — square brackets are rewritten by the text normalizer ([/] → '), so [sigh] reaches the model as 'sigh'. The only inline markers that survive tokenization are the uppercase <|…|> special tokens inherited from the ASR annotation vocabulary: <|Laughter|>, <|Applause|>, <|BGM|>, <|HAPPY|>, <|SAD|>, <|ANGRY|>, <|NEUTRAL|>.

These are not documented generative controls — empirically <|Laughter|> injects an unpredictable non-speech vocalization rather than a clean laugh. In the upstream model, real emotion control uses separate inputs (emo_vector, emo_audio_prompt, emo_text); this port intentionally drops those and lets the reference audio carry both timbre and emotion. So the reliable way to get an expressive read (a sigh, a giggle, an excited tone) is to put that expression in the reference clip.

Install / one-click run

No install needed with uv:

uvx index-tts-2.5-mlx synth \
    --ref /path/to/voice.wav \
    --text "大家好, this is IndexTTS running on MLX." \
    --out out.wav

The first run downloads the int8 model (~5 GB) from Hugging Face into the standard HF cache (~/.cache/huggingface); later runs reuse it. Each synth prints a timing report:

wrote out.wav
audio       2.83 s
load        2.12 s   (model download + weight load)
clone       1.69 s   (speaker embedding from --ref)
synth       1.32 s
RTF        0.467     (2.14x realtime; <1 = faster than real-time)
stages    gpt=0.35s  codec=0.00s  regulator=0.00s  cfm=0.46s  bigvgan=0.51s

(load is only slow the very first time, while it downloads. synth is the marginal cost per line once warm — reuse one spk across lines to skip repeated clone work.)

Or install into an environment:

pip install index-tts-2.5-mlx

Pre-download the weights ahead of time:

uvx index-tts-2.5-mlx download

CLI usage

index-tts-2.5-mlx synth --ref REF.wav --text "..." --out out.wav [options]

Option	Default	Description
--ref	(required)	Reference audio to clone (≤15 s, clear speech).
--text	(required)	Text to synthesize (zh/en/ja/yue, mixed OK).
--out	output.wav	Output WAV path (22050 Hz, int16).
--lang	zh	Language hint: zh, en, ja, yue.
--greedy	off	Greedy decoding (deterministic).
--seed	random	RNG seed for sampling.
--top-k / --top-p / --temperature	30 / 0.8 / 0.8	Sampling controls.
--repetition-penalty	10.0	Repetition penalty.
--max-mel-tokens	1500	Max acoustic tokens per segment.
--duration-factor	1.0	Speech-rate multiplier.
--n-timesteps / --cfg-rate	25 / 0.7	Flow-matching solver controls.
--model-dir	auto	Use a local weight dir instead of downloading.
--no-normalization	off	Disable text normalization.

Python API

from index_tts_2_5_mlx import IndexTTS

tts = IndexTTS()                       # auto-downloads weights on first use
sr, pcm = tts.clone(
    "AI 模型在 2025 年处理了 100 万条数据。",
    ref_audio_path="voice.wav",
    out="clone.wav",                   # optional; also returns pcm
    lang="zh",
)

# Reuse one cloned voice across many lines (build the speaker once):
spk = tts.build_speaker("voice.wav")
for i, line in enumerate(["第一句。", "Second sentence.", "第三句。"]):
    sr, pcm = tts.clone(line, ref_audio_path=None, spk=spk, out=f"line{i}.wav")

synthesize(...) returns the raw int16 PCM array (numpy) at tts.sample_rate (22050 Hz); clone(...) additionally writes a WAV when out is given. Use your own reference audio only with permission — see License.

Speed

End-to-end synthesis, warm, mean of 3 runs (Apple M5 Pro). RTF = synthesis time ÷ audio duration (lower is better; <1 = faster than real-time).

Backend	fx0 RTF	fx1 RTF	vs PyTorch MPS
PyTorch MPS (official)	1.17	1.11	1.0×
MLX fp32	0.67	0.71	~1.7×
MLX int8 (this package)	0.47	0.45	~2.4×

Stage breakdown (int8, ~3 s of audio): GPT decode ≈ 0.35 s, flow-matching CFM ≈ 0.48 s, BigVGAN vocoder ≈ 0.56 s. The int8 quantization fuses dequant into the Metal matmul kernels, which is where the GPT autoregressive decode speedup comes from; the other modules are compute-bound and stay fp32.

MLX vs PyTorch MPS on Apple Silicon

Both MLX and PyTorch-MPS run on the same Metal GPU, so why is MLX ~2.4× faster here?

• Unified memory, zero copies. PyTorch-MPS keeps a host/device split and pays CPU↔GPU transfer and synchronization costs across the pipeline's many small ops. MLX targets Apple Silicon's unified memory directly — arrays live in one address space, so the numpy orchestration (frontend, samplers, solvers) and the GPU networks hand off without copies.
• Fused Metal kernels. MLX fuses common subgraphs and, crucially, fuses int8 dequantization into the matmul kernel (nn.QuantizedLinear). The GPT autoregressive decode is memory-bandwidth-bound, so reading 1-byte weights instead of 4-byte ones nearly doubles decode throughput (GPT: ~0.66 s fp32 → ~0.35 s int8).
• Lighter runtime. No PyTorch dispatcher / autograd overhead on the inference path.

A note on the baseline: the torch-MPS backend itself diverges from the torch-CPU reference under greedy decoding (MPS matmul rounding), so MPS numbers are a speed reference, not an accuracy reference. This MLX port is verified against the deterministic CPU fixtures, not against MPS.

For context, a CPU-only MNN build of the same pipeline lands at RTF ≈ 2.4 (fp32) — and its int8/int4 weight-only quantization actually runs slower than fp32, because MNN dequantizes weights back to fp32 at load time and computes in fp32. The bottleneck (the CFM DiT) is FLOP-bound, where the GPU is ~12× faster than CPU; no CPU quantization closes that. MLX on the GPU is the right target for Apple Silicon.

Quality / effect

Numeric fidelity — each MLX module matches the PyTorch reference with cosine similarity ≥ 0.999; the full greedy pipeline reproduces the reference acoustic tokens exactly (fx0 73/73, fx1 83/83). Vocoder output matches the reference at mel-spectrogram SNR ≥ 25 dB (inaudible difference).

Voice cloning — measured with a CampPlus speaker-embedding cosine between each synthesized clip and its reference vs. an unrelated voice. Every clip scores higher against its own reference, confirming the timbre follows the given reference:

Clip	sim(own ref)	sim(other voice)	follows ref
clone A ×3	0.61–0.73	0.29–0.39	✓
clone B ×3	0.51–0.61	0.48–0.50	✓

Intelligibility (ASR, Whisper) — synthesized mixed-language clips with numbers and abbreviations transcribe correctly, e.g. AI 模型在 2025 年处理了 100 万条数据。 → “AI…2025 年处理了 100 万条数据”, and Use the CPU or GPU, 都可以 → “用 CPU 或 GPU 都可以”. Cloning quality tracks reference quality: use a clean, natural recording.

How it works

Pipeline: text frontend (tiktoken + wetext normalization) → w2v-bert semantic features → semantic codec → int8 GPT autoregressive acoustic tokens → length regulator → flow-matching CFM (DiT) → BigVGAN vocoder → 22050 Hz WAV. The orchestration (samplers, solvers, DSP) is numpy; the heavy networks are MLX modules loading the shipped safetensors.

Model weights: yunfengwang/IndexTTS-2.5-mlx (int8 GPT + fp32 feed-forward modules).

PyTorch → MLX conversion notes

For anyone porting a similar model, this is what the conversion involved. Eight sub-networks were re-implemented as MLX nn.Modules and verified against PyTorch reference dumps.

Weight extraction. Each torch state_dict is dumped to safetensors, then remapped to MLX layout:

• Everything runs channels-last [B, T, C] (vs torch [B, C, T]).
• Conv1d weight [out, in, K] → MLX [out, K, in] via transpose(0, 2, 1).
• ConvTranspose1d weight [in, out, K] → MLX [out, K, in] via transpose(1, 2, 0).
• Conv2d weight [out, in, kh, kw] → MLX [out, kh, kw, in].
• nn.Linear weight [out, in] copies as-is.
• weight_norm (g/v) is folded into a single dense weight at export time.
• GroupNorm needs pytorch_compatible=True; BatchNorm modules need .eval() (MLX defaults to training mode and would otherwise use batch statistics instead of the running stats).

Gotchas hit during the port:

• GPU conv precision. MLX's GPU conv_transpose1d (stride > 1) and conv1d with non-multiple-of-16 channels accumulate at reduced precision, which wrecked the BigVGAN vocoder phase. Fixed by rewriting the transpose-conv as a phase-decomposed stride-1 conv1d and padding channels to a multiple of 16 in fp32 — lifting vocoder SNR from ~12 dB to ~89 dB against the torch reference.
• Relative-position attention. The w2v-bert semantic encoder uses relative_key attention: a per-layer distance_embedding table (73 buckets = 64 left + 8 right + 1, dim 64), with an additive bias einsum("bhld,lrd->bhlr", q, emb[clip(r-l,-64,8)+64]) / 8 fed into mx.fast.scaled_dot_product_attention. Only 17 of the 24 layers actually execute (hidden_states[17]).
• State in modules. An mx.array can't be a plain nn.Module instance attribute; lazy module-level globals are used instead.
• Sample-vs-mel SNR. A tiny GPU-vs-CPU CFM difference (mel cos ≈ 0.999991) is amplified by the vocoder into fine phase detail, so raw sample-SNR looks low (~12 dB) while the mel-domain SNR (≈33 dB) is inaudible. Fidelity is therefore gated in the mel domain.

int8 quantization. Only the GPT is quantized — it's the decode bottleneck and the only memory-bandwidth-bound module (the CFM/BigVGAN are FLOP-bound, so quant doesn't help them). nn.quantize(group_size=64, bits=8, class_predicate=Linear) quantizes the Linear mats while keeping embeddings in fp32. For distribution the quantized weights are serialized (packed weight + scales + biases) into gpt_int8.safetensors (909 MB vs 2.3 GB fp32); at load time a pre-quantized Gpt is built first, then the packed params load straight into it (load_weights(..., strict=True)) — no fp32 detour, and the round-trip is bit-identical.

Verification. Every module is gated at cosine ≥ 0.999 against a deterministic torch fixture before being wired in; the end-to-end check reproduces the reference greedy acoustic tokens exactly (fx0 73/73, fx1 83/83 — the MPS baseline manages only ~1.4% on fx1).

License

Code in this package is provided under the same terms as the upstream project. IndexTTS-2.5 model weights are subject to the original Bilibili IndexTTS license — see the upstream model card. Use voice cloning responsibly and only with consent from the voice owner.