mlx-speech

[!NOTE] This project wouldn't exist without the inspiration and generous support of the incredible community at linux.do.

Local speech synthesis, editing, and transcription on Apple Silicon, running pure MLX. No cloud, no PyTorch.

mlx-speech is an App Automaton project. Project page: appautomaton.github.io/mlx-speech. The appautomaton org hosts the code on GitHub and the converted weights on Hugging Face.

Models

Pre-converted MLX weights are published under the App Automaton Hugging Face org, appautomaton, and download automatically on first use. Flat model repositories load by alias or full repo id — tts.load("fish-s2-pro") and tts.load("appautomaton/fishaudio-s2-pro-8bit-mlx") are equivalent. Shared multi-artifact repositories use an alias or an explicit artifact_subdir so the runtime never guesses a variant. Each model name links to a guide covering behavior, flags, and known limitations.

Text-to-speech

Speech-to-text

¹ tts.load("dramabox") also pulls the Gemma 3 12B backbone text encoder automatically. Output is 48 kHz stereo. For advanced controls (cfg, steps, voice reference) use scripts/generate_dramabox.py. Optional denoise_ref=True cleans a noisy voice reference with the pure-MLX RE-USE / SEMamba enhancer (off by default; NSCLv1 non-commercial weights). See docs/dramabox.md.

Installation

Requires an Apple Silicon Mac (M1 or later) and Python 3.13+.

pip install mlx-speech

Quick Start

Python:

import mlx_speech

# Text-to-speech
model = mlx_speech.tts.load("fish-s2-pro")
result = model.generate("Hello from mlx-speech!")
# result.waveform: mx.array, result.sample_rate: int

# Voice cloning with emotion tags
result = model.generate(
    "[excited] This is amazing!",
    reference_audio="reference.wav",
    reference_text="Transcript of the reference audio.",
)

# Speech-to-text
asr = mlx_speech.asr.load("qwen3-asr-1.7b")
print(asr.generate("audio.wav").text)

# True cache-aware waveform streaming is available on Nemotron
nemotron = mlx_speech.asr.load("nemotron-asr-streaming")
session = nemotron.stream_session(language="en-US", att_context_size=(56, 3))

# Local checkpoint paths work anywhere an alias does
granite = mlx_speech.asr.load("models/ibm/granite_4_0_1b_speech/original")
print(granite.generate("audio.wav").text)

# Discover models
mlx_speech.tts.list_models()
mlx_speech.tts.list_models(detailed=True)  # includes shared-repo artifact paths
mlx_speech.asr.list_models()

CLI:

# Generate speech
mlx-speech tts --model fish-s2-pro --text "Hello!" -o output.wav

# Bounded waveform streaming with dots.tts
mlx-speech tts --model dots-tts-soar --text "Hello!" --stream -o streamed.wav

# Voice cloning with emotion tags
mlx-speech tts --model fish-s2-pro \
  --text "[whisper] Just between us..." \
  --reference-audio ref.wav \
  --reference-text "Transcript of reference." \
  -o cloned.wav

# Step Audio emotion editing
mlx-speech tts --model step-audio \
  --reference-audio input.wav \
  --reference-text "Transcript." \
  --edit-type emotion --edit-info happy \
  -o happy.wav

# Sound effect generation
mlx-speech tts --model moss-sound-effect \
  --text "rolling thunder with rainfall" \
  --duration-seconds 8 \
  -o thunder.wav

# Transcribe audio
mlx-speech asr --model cohere-asr --audio speech.wav
mlx-speech asr --model qwen3-asr-1.7b --audio speech.wav --language Chinese
mlx-speech asr --model nemotron-asr-streaming --audio speech.wav --language en-US

# Local checkpoint paths work anywhere an alias does
mlx-speech tts --model models/fish_s2_pro/mlx-int8 --text "Hello!" -o output.wav
mlx-speech asr --model models/ibm/granite_4_0_1b_speech/original --audio speech.wav

# Discover models
mlx-speech tts --list-models
mlx-speech asr --list-models
mlx-speech --help

Note: The mlx-speech CLI covers the common path — basic generation, voice cloning, and editing. For advanced controls (sampling temperature, top-p/k, diffusion steps, batch JSONL, duration tuning, etc.) use the scripts in scripts/ directly. Each model family has a corresponding script with the full inference surface documented in docs/.

Conversion

To convert upstream source weights yourself:

python scripts/convert/fish_s2_pro.py
python scripts/convert/longcat_audiodit.py
python scripts/convert/vibevoice.py
python scripts/convert/moss_local.py
python scripts/convert/moss_ttsd.py
python scripts/convert/moss_sound_effect.py
python scripts/convert/step_audio_editx.py
python scripts/convert/cohere_asr.py
python scripts/convert/qwen3_asr.py

Development

git clone https://github.com/appautomaton/mlx-speech.git
cd mlx-speech
uv sync
uv run pytest tests/unit/
uv run ruff check .

mlx-speech/
  src/mlx_speech/    library code
  scripts/           conversion, generation, eval, and audit entry points
  models/            local checkpoints (not in git)
  tests/             unit, checkpoint, runtime, integration tests
  docs/              model-family behavior guides

License

MIT — see LICENSE

Built and maintained by App Automaton.