mlx-speech 0.4.1

MLX-native speech library for Apple Silicon.

mlx-speech

[!NOTE] This project wouldn't exist without the inspiration and generous support of the incredible community at linux.do.

Local speech synthesis, editing, and transcription on Apple Silicon, running pure MLX. No cloud, no PyTorch.

Alias	Type	Description
fish-s2-pro	TTS	Fish S2 Pro — dual-AR TTS, voice cloning, emotion tags
vibevoice	TTS	VibeVoice Large — hybrid LLM+diffusion TTS, voice cloning
longcat	TTS	LongCat AudioDiT — flow-matching diffusion TTS
moss-local	TTS	OpenMOSS TTS Local — local-attention multi-VQ TTS
moss-ttsd	TTS	OpenMOSS TTS Delay — delay-pattern dialogue TTS
moss-sound-effect	TTS	OpenMOSS Sound Effect — text-to-sound-effect generation
step-audio	TTS	Step-Audio-EditX — voice cloning + audio editing
dramabox	TTS	DramaBox — Resemble flow-matching diffusion TTS (48 kHz stereo, LTX-2 derived)
cohere-asr	ASR	Cohere Transcribe — multilingual ASR
local Granite Speech path	ASR	IBM Granite Speech 4.0 1B — local-path ASR
local Qwen3-ASR path	ASR	Qwen3-ASR-1.7B — English, Chinese, and mixed Chinese/English local-path ASR

Requirements

• Apple Silicon Mac (M1 or later)
• Python 3.13+

Installation

pip install mlx-speech

Quick Start

Models download automatically from HuggingFace on first use.

Python API:

import mlx_speech

# Text-to-speech
model = mlx_speech.tts.load("fish-s2-pro")
result = model.generate("Hello from mlx-speech!")
# result.waveform: mx.array, result.sample_rate: int

# Voice cloning with emotion tags
result = model.generate(
    "[excited] This is amazing!",
    reference_audio="reference.wav",
    reference_text="Transcript of the reference audio.",
)

# Speech-to-text
asr = mlx_speech.asr.load("cohere-asr")
result = asr.generate("audio.wav")
print(result.text)

# Local Granite Speech checkpoint
granite = mlx_speech.asr.load("models/ibm/granite_4_0_1b_speech/original")
print(granite.generate("audio.wav").text)

# Local Qwen3-ASR BF16 checkpoint
qwen = mlx_speech.asr.load("models/Qwen3-ASR-1.7B-MLX-BF16")
print(qwen.generate("audio.wav").text)

# List available models
mlx_speech.tts.list_models()
mlx_speech.asr.list_models()

CLI:

# Generate speech
mlx-speech tts --model fish-s2-pro --text "Hello!" -o output.wav

# Voice cloning with emotion tags
mlx-speech tts --model fish-s2-pro \
  --text "[whisper] Just between us..." \
  --reference-audio ref.wav \
  --reference-text "Transcript of reference." \
  -o cloned.wav

# Step Audio emotion editing
mlx-speech tts --model step-audio \
  --reference-audio input.wav \
  --reference-text "Transcript." \
  --edit-type emotion --edit-info happy \
  -o happy.wav

# Sound effect generation
mlx-speech tts --model moss-sound-effect \
  --text "rolling thunder with rainfall" \
  --duration-seconds 8 \
  -o thunder.wav

# Transcribe audio
mlx-speech asr --model cohere-asr --audio speech.wav
mlx-speech asr --model models/ibm/granite_4_0_1b_speech/original --audio speech.wav
mlx-speech asr --model models/Qwen3-ASR-1.7B-MLX-BF16 --audio speech.wav
mlx-speech asr --model models/Qwen3-ASR-1.7B-MLX-BF16 --audio speech.wav --language Chinese

# Discover models
mlx-speech tts --list-models
mlx-speech asr --list-models
mlx-speech --help

Local model paths work too:

mlx-speech tts --model models/fish_s2_pro/mlx-int8 --text "Hello!" -o output.wav

Note: The mlx-speech CLI covers the common path — basic generation, voice cloning, and editing. For advanced controls (sampling temperature, top-p/k, diffusion steps, batch JSONL, duration tuning, etc.) use the scripts in scripts/ directly. Each model family has a corresponding script with the full inference surface documented in docs/.

Models

Pre-converted MLX weights are on Hugging Face under appautomaton. Use mlx_speech.tts.load("alias") or mlx_speech.tts.load("appautomaton/repo-name") to load them.

Alias	HF Repo	Quant
fish-s2-pro	fishaudio-s2-pro-8bit-mlx	int8
vibevoice	vibevoice-mlx	int8
longcat	longcat-audiodit-3.5b-8bit-mlx	int8
moss-local	openmoss-tts-local-mlx	int8
moss-ttsd	openmoss-ttsd-mlx	int8
moss-sound-effect	openmoss-sound-effect-mlx	4-bit
step-audio	step-audio-editx-8bit-mlx	int8
cohere-asr	cohere-asr-mlx	int8

Conversion

Convert from upstream source weights:

python scripts/convert/fish_s2_pro.py
python scripts/convert/longcat_audiodit.py
python scripts/convert/vibevoice.py
python scripts/convert/moss_local.py
python scripts/convert/moss_ttsd.py
python scripts/convert/cohere_asr.py
python scripts/convert/qwen3_asr.py

Model Guides

Each family has a doc covering behavior, flags, and known limitations:

• Fish S2 Pro
• LongCat AudioDiT
• MossTTSLocal
• MOSS-TTSD
• MOSS-SoundEffect
• VibeVoice
• Step-Audio-EditX
• DramaBox
• CohereASR
• Granite Speech ASR
• Qwen3-ASR

Development

git clone https://github.com/appautomaton/mlx-speech.git
cd mlx-speech
uv sync
uv run pytest tests/unit/
uv run ruff check .

mlx-speech/
  src/mlx_speech/    library code
  scripts/           conversion, generation, eval, and audit entry points
  models/            local checkpoints (not in git)
  tests/             unit, checkpoint, runtime, integration tests
  docs/              model-family behavior guides

License

MIT — see LICENSE