Chatterbox MLX - Apple Silicon Optimized TTS

An MLX-optimized fork of Resemble AI's Chatterbox TTS for Apple Silicon, delivering up to 2.4x faster inference.

Installation

pip install chatterbox-mlx

Requirements

• macOS with Apple Silicon (M1/M2/M3/M4)
• Python 3.11+ (tested primarily with 3.11.12. Also tested with 3.12.12)
• ~4GB disk space for model weights

Important: Python must be compiled with lzma support. If you're using pyenv:

# Install xz library first (provides liblzma)
brew install xz

# Then install Python (or reinstall if already installed)
pyenv install 3.11.12  # or your preferred version

If you see an error about ModuleNotFoundError: No module named '_lzma', you need to install xz and reinstall Python.

CLI Usage

Generate speech directly from the terminal:

    # Generate English speech (auto-generated filename):
    chatterbox "Artificial intelligence has made remarkable strides in recent years, particularly in the field of natural language processing."

    # Generate Spanish speech:
    chatterbox "La inteligencia artificial ha logrado avances notables en los últimos años." --lang es

    # Use the --voice flag to provide a reference audio file for voice cloning:
    chatterbox "Artificial intelligence has made remarkable strides in recent years, particularly in the field of natural language processing." --voice speaker.wav

    # Run multilingual benchmark (saves to benchmark_output/)
    chatterbox --benchmark --languages en es

CLI Options

Quick Start

import torchaudio as ta
from chatterbox.tts_mlx import ChatterboxTTSMLX

# Load model (downloads weights automatically on first run). T3 runs on MLX;
# device sets where S3Gen runs: "mps" (default, Apple GPU) or "cpu".
model = ChatterboxTTSMLX.from_pretrained(device="mps")


# Generate speech
text = "Hello! This is Chatterbox running with MLX optimization on Apple Silicon."
wav = model.generate(text)
ta.save("output.wav", wav, model.sr)

# Voice cloning with reference audio
wav = model.generate(
    text,
    audio_prompt_path="reference_voice.wav",
    exaggeration=0.5,  # Emotion intensity (0.0-1.0)
    cfg_weight=0.5,    # Classifier-free guidance
)

Long-Form Audio Generation

For texts longer than ~50 words, use chunked generation:

long_text = """
Your long text here. It can span multiple paragraphs and sentences.
The generate_long method will automatically split it at sentence boundaries,
generate each chunk separately, and crossfade them together seamlessly.
"""

wav = model.generate_long(
    long_text,
    audio_prompt_path="reference_voice.wav",
    chunk_size_words=50,
    overlap_duration=0.1,
)
ta.save("long_output.wav", wav, model.sr)

Chatterbox-Nano and Chatterbox-Turbo (fastest, English)

Chatterbox-Turbo (350M) and Chatterbox-Nano (110M) use a GPT-2 T3 decoder and the meanflow S3Gen vocoder. Both models use one class. This port runs the T3 decode loop, the sampling and the vocoder in MLX.

import soundfile as sf
from chatterbox.tts_turbo_mlx import ChatterboxTurboMLX

model = ChatterboxTurboMLX.from_pretrained(nano=True)  # nano=False loads Turbo
# Optional: clone a voice (longer than 5 s). Without it, the built-in voice is used.
model.prepare_conditionals("reference_voice.wav")

wav = model.generate("Hello! [chuckle] This is Chatterbox-Nano on Apple Silicon.", seed=0)
sf.write("nano.wav", wav, model.sr)
print(model.last_timings)  # t3_s, s3gen_s, watermark_s, n_speech_tokens, n_chunks, ...

generate() splits long text at sentence boundaries. It decodes up to 8 chunks together as one batch, then crossfades them. Each chunk has a token limit from its word count, so one chunk cannot run on. The output has a Perth watermark, the same as upstream. Set watermark=False to skip it.

Measured on an Apple M4 (32 GB), fp16. RTF is generation time divided by audio time, so lower is faster. WER comes from Whisper large-v3-turbo. WER ranges cover three LibriSpeech voices (two female, one male), the built-in voice (Nano and Turbo only), and two seeds.

For the 292-word text, the WER is the mean over the sentence chunks. Whisper adds repeated words to clips longer than about 30 s, so a WER of the full clip is not reliable.

Useful options:

• quantize_bits (load time): 8 makes Turbo about 30% faster on medium text. 4 makes Nano about 25% faster on medium text. Do not use 4 with Turbo: long text runs on and its WER goes up. The quantization WER checks used a synthetic reference only. Listen to the output before you use quantization.
• max_batch_chunks (default 8): set 1 to decode chunks one by one. Long text is then 1.5-2x slower.
• release_caches(): frees the KV caches. The next call rebuilds them.

Known limits:

• Use a reference clip of clear speech, 10-15 s long. A non-speech clip (a tone or music) makes the model babble at the start of each sentence chunk.
• cfg_weight, exaggeration and min_p have no effect, the same as upstream Turbo.

Chatterbox-Flash (English)

Chatterbox-Flash replaces the autoregressive T3 decoder with a block-diffusion decoder. It unmasks a block of speech tokens in parallel per step. This port runs the full T3 loop and the meanflow S3Gen vocoder in MLX.

import soundfile as sf
from chatterbox.tts_flash_mlx import ChatterboxFlashMLX

model = ChatterboxFlashMLX.from_pretrained()  # MLX vocoder by default
model.prepare_conditionals("reference_voice.wav")  # required: Flash has no built-in voice

wav = model.generate("Hello! This is Chatterbox-Flash running on Apple Silicon.", seed=0)
sf.write("flash.wav", wav, model.sr)
print(model.last_timings)  # t3_s, s3gen_s, n_speech_tokens, n_chunks, ...

generate() splits text over 35 words at sentence boundaries. It decodes up to 4 chunks together as one batch, then crossfades them. You do not need a separate long-form method.

Measured on an Apple M4 (32 GB), fp16. RTF is generation time divided by audio time, so lower is faster. The WER is the range over three LibriSpeech voices and two seeds, from Whisper large-v3-turbo.

Useful options:

• block_size (load time, default 24): diffusion block length. Set 16 for the paper configuration. Set 32 for about 20% more speed at a small WER cost.
• preset ("paper", "balanced", "fast"): early-decoding strength. "paper" is the default. Stronger presets use fewer steps per block, at some quality cost on short text.
• max_batch_chunks (default 4): set 1 to decode chunks one by one.
• release_caches(): frees the KV caches after a long document. The next call rebuilds them.

Known limits:

• Keep cfg_scale=1.0. Without guidance, long text loops and the pitch drifts.
• With some voices, Flash adds a few babbled words after the end of a short text (WER 0.23 on medium text with one of three test voices).
• 4-bit quantization breaks long text. Do not use it with Flash.
• Flash output has no watermark, the same as upstream Flash.

🙏 Acknowledgements

This project is built on top of the excellent Chatterbox TTS by Resemble AI. I'm deeply grateful for their work in creating and open-sourcing a production-grade, multilingual text-to-speech system under the MIT license.

This fork focuses specifically on MLX optimizations for Apple Silicon. If you're looking for the original project with CUDA support and the full feature set, please visit the official Resemble AI repository.

What's Different in This Fork?

This package provides native MLX acceleration for Apple Silicon Macs, achieving significant performance improvements:

Key Optimizations

• MLX-Native T3 Model: The 520M parameter Llama 3 backbone runs entirely on MLX
• Float16 KV Cache: Up to 5.8 GB memory savings with 32% faster generation
• Hybrid Architecture: Combines MLX speed with PyTorch quality controls
• Long-Form Generation: Intelligent chunking with crossfade for extended audio

Benchmark Results

All benchmarks run on Apple M4 (32GB RAM), macOS 15.4, Python 3.11, PyTorch 2.8.0.

English TTS Performance

Key findings:

• Hybrid-MLX recommended for production (best quality/speed balance)
• Pure MLX fastest for short texts, but quality degrades on long texts
• 2.2-2.4x speedup vs CPU baseline across all text lengths

Multilingual Performance

Visual Comparison

                    GENERATION TIME COMPARISON

     Short (5 words)
     ├─ CPU        ████████████████████████████████████████ 8.91s
     ├─ Hybrid-MLX ██████████████████ 4.08s (2.2x faster)
     └─ Pure MLX   ████████████████ 3.70s (2.4x faster)

     Medium (31 words)
     ├─ CPU        ████████████████████████████████████████ 57.51s
     ├─ Hybrid-MLX █████████████████ 25.24s (2.3x faster)
     └─ Pure MLX   ████████████████ 24.40s (2.4x faster)

     Long (94 words)
     ├─ CPU        ████████████████████████████████████████ 137.92s
     ├─ Hybrid-MLX █████████████████ 62.66s (2.2x faster)  ✓ Best quality
     └─ Pure MLX   ██████████████████ 68.82s (2.0x faster)

Backend Comparison

RTF = Real-Time Factor (audio_duration / generation_time). Higher is better.

Running Benchmarks

You can reproduce these benchmarks on your own hardware.

English TTS Benchmark

# Full benchmark (all backends)
python benchmark_mps.py --runs 3 --validate

# Quick test with Hybrid-MLX only
python benchmark_mps.py --hybrid-mlx-only --runs 1

# CPU baseline only
python benchmark_mps.py --cpu-only --runs 1

# With voice cloning
python benchmark_mps.py --audio-prompt speaker.wav --runs 3

# Enable memory debugging
DEBUG_MEMORY=1 python benchmark_mps.py --hybrid-mlx-only

Options:

Multilingual Benchmark

# Test specific languages
python benchmark_multilingual.py \
    --audio-prompt speaker.wav \
    --languages en es fr de ja zh \
    --runs 3

# Quick test with Hybrid-MLX
python benchmark_multilingual.py \
    --audio-prompt speaker.wav \
    --languages en es \
    --hybrid-mlx-only

# With validation
python benchmark_multilingual.py \
    --audio-prompt speaker.wav \
    --languages en es fr \
    --validate

Supported Languages: en (English), es (Spanish), fr (French), de (German), it (Italian), pt (Portuguese), ru (Russian), ja (Japanese), zh (Chinese), ko (Korean), ar (Arabic), hi (Hindi), tr (Turkish), pl (Polish), nl (Dutch), sv (Swedish), da (Danish), no (Norwegian), fi (Finnish), el (Greek), he (Hebrew), ms (Malay), sw (Swahili)

Benchmark Output

Results are saved to:

• benchmark_output/benchmark_results.json - English TTS results
• benchmark_multilingual_output/multilingual_results.json - Multilingual results
• Generated audio files: {device}_{category}.wav

Architecture

Chatterbox is a two-stage TTS pipeline. This fork accelerates the most compute-intensive component (T3) with MLX:

┌─────────────────────────────────────────────────────────────────────┐
│                     CHATTERBOX MLX PIPELINE                         │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────────────┐   │
│  │ VoiceEncoder │    │     T3       │    │       S3Gen          │   │
│  │  (PyTorch)   │───▶│    (MLX)     │───▶│   (PyTorch/MPS)      │   │
│  │   ~2M params │    │  520M params │    │     ~80M params      │   │
│  └──────────────┘    └──────────────┘    └──────────────────────┘   │
│                            ▲                                        │
│                            │                                        │
│                    2.4x faster with MLX                             │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Supported Languages

All 23 languages from the original Chatterbox are supported:

Arabic • Danish • German • Greek • English • Spanish • Finnish • French • Hebrew • Hindi • Italian • Japanese • Korean • Malay • Dutch • Norwegian • Polish • Portuguese • Russian • Swedish • Swahili • Turkish • Chinese

from chatterbox.mtl_tts_mlx import ChatterboxMultilingualTTSMLX

model = ChatterboxMultilingualTTSMLX.from_pretrained(device="mps")

# French
wav = model.generate("Bonjour, comment ça va?", language_id="fr")

# Japanese
wav = model.generate("こんにちは、元気ですか？", language_id="ja")

Tips for Best Results

General Use

• Default settings (exaggeration=0.5, cfg_weight=0.5) work well for most cases
• Ensure reference audio matches target language to avoid accent transfer

Expressive Speech

• Lower cfg_weight (~0.3) + higher exaggeration (~0.7) for dramatic delivery
• Higher exaggeration speeds up speech; lower cfg_weight compensates

Memory Usage

Enable debug logging to monitor memory:

DEBUG_MEMORY=1 python your_script.py

Differences from Original Chatterbox

Credits & Links

• Original Project: Resemble AI's Chatterbox
• Resemble AI: resemble.ai - For creating and open-sourcing this incredible TTS system
• Demo: Hugging Face Space
• Evaluation: Outperforms ElevenLabs

Upstream Dependencies

• CosyVoice
• HiFT-GAN
• Llama 3
• MLX

License

MIT License - Same as the original Chatterbox project.

Citation

If you use this project, please cite the original Chatterbox:

@misc{chatterboxtts2025,
  author       = {{Resemble AI}},
  title        = {{Chatterbox-TTS}},
  year         = {2025},
  howpublished = {\url{https://github.com/resemble-ai/chatterbox}},
  note         = {GitHub repository}
}