mlx-speech 0.2.0

MLX-native speech library for Apple Silicon.

mlx-speech

[!NOTE] This project wouldn't exist without the inspiration and generous support of the incredible community at linux.do.

Local speech synthesis, editing, and transcription on Apple Silicon, running pure MLX. No cloud, no PyTorch.

Alias	Type	Description
fish-s2-pro	TTS	Fish S2 Pro — dual-AR TTS, voice cloning, emotion tags
vibevoice	TTS	VibeVoice Large — hybrid LLM+diffusion TTS, voice cloning
longcat	TTS	LongCat AudioDiT — flow-matching diffusion TTS
moss-local	TTS	OpenMOSS TTS Local — local-attention multi-VQ TTS
moss-ttsd	TTS	OpenMOSS TTS Delay — delay-pattern dialogue TTS
cohere-asr	ASR	Cohere Transcribe — multilingual ASR

Requirements

• Apple Silicon Mac (M1 or later)
• Python 3.13+

Installation

pip install mlx-speech

Quick Start

Models download automatically from HuggingFace on first use.

Python API:

import mlx_speech

# Text-to-speech
model = mlx_speech.tts.load("fish-s2-pro")
result = model.generate("Hello from mlx-speech!")
# result.waveform: mx.array, result.sample_rate: int

# Voice cloning with emotion tags
result = model.generate(
    "[excited] This is amazing!",
    reference_audio="reference.wav",
    reference_text="Transcript of the reference audio.",
)

# Speech-to-text
asr = mlx_speech.asr.load("cohere-asr")
result = asr.generate("audio.wav")
print(result.text)

# List available models
mlx_speech.tts.list_models()
mlx_speech.asr.list_models()

CLI:

# Generate speech
mlx-speech-tts --model fish-s2-pro --text "Hello!" -o output.wav

# Voice cloning
mlx-speech-tts --model fish-s2-pro \
  --text "[whisper] Just between us..." \
  --reference-audio ref.wav \
  --reference-text "Transcript of reference." \
  -o cloned.wav

# Transcribe audio
mlx-speech-asr --model cohere-asr --audio speech.wav

# List available models
mlx-speech-tts --list-models
mlx-speech-asr --list-models

Local model paths work too:

mlx-speech-tts --model models/fish_s2_pro/mlx-int8 --text "Hello!" -o output.wav

Models

Pre-converted MLX weights are on Hugging Face under appautomaton. Use mlx_speech.tts.load("alias") or mlx_speech.tts.load("appautomaton/repo-name") to load them.

Alias	HF Repo	Quant
fish-s2-pro	fishaudio-s2-pro-8bit-mlx	int8
vibevoice	vibevoice-mlx	int8
longcat	longcat-audiodit-3.5b-8bit-mlx	int8
moss-local	openmoss-tts-local-mlx	int8
moss-ttsd	openmoss-ttsd-mlx	int8
cohere-asr	cohere-asr-mlx	int8

Additional models available via scripts/:

Model	Script	Notes
MOSS-SoundEffect	scripts/generate_moss_sound_effect.py	Text-to-sound-effect
Step-Audio-EditX	scripts/generate_step_audio_editx.py	Voice cloning and audio editing

Conversion

Convert from upstream source weights:

python scripts/convert_fish_s2_pro.py
python scripts/convert_longcat_audiodit.py
python scripts/convert_vibevoice.py
python scripts/convert_moss_local.py
python scripts/convert_moss_ttsd.py
python scripts/convert_cohere_asr.py

Model Guides

Each family has a doc covering behavior, flags, and known limitations:

• Fish S2 Pro
• LongCat AudioDiT
• MossTTSLocal
• MOSS-TTSD
• MOSS-SoundEffect
• VibeVoice
• Step-Audio-EditX
• CohereASR

Development

git clone https://github.com/appautomaton/mlx-speech.git
cd mlx-speech
uv sync
uv run pytest tests/unit/
uv run ruff check .

mlx-speech/
  src/mlx_speech/    library code
  scripts/           conversion and generation entry points
  models/            local checkpoints (not in git)
  tests/             unit, checkpoint, runtime, integration tests
  docs/              model-family behavior guides

License

MIT — see LICENSE