LuxTTS

   

LuxTTS is an lightweight zipvoice based text-to-speech model designed for high quality voice cloning and realistic generation at speeds exceeding 150x realtime.

https://github.com/user-attachments/assets/a3b57152-8d97-43ce-bd99-26dc9a145c29

Upstream sync status (2026-02-11)

• Upstream LuxTTS commit synced: 34820963ee97f406619e5983771e572f779a600a (2026-01-28).
• Local MLX fork is 0 commits behind upstream on master.
• Ongoing MLX quality and parity plan: docs/MLX_PORT_PLAN.md.

Release status

• Initial stable package release: v0.1.0 (2026-02-13).
• Release notes: CHANGELOG.md.

The main features are

• Voice cloning: SOTA voice cloning on par with models 10x larger.
• Clarity: Clear 48khz speech generation unlike most TTS models which are limited to 24khz.
• Speed: Reaches speeds of 150x realtime on a single GPU and faster then realtime on CPU's as well.
• Efficiency: Fits within 1gb vram meaning it can fit in any local gpu.

Usage

You can try it locally, colab, or spaces.

Simple installation:

pip install LuxTTS-mlx

Required vocoder dependency (LinaCodec):

pip install git+https://github.com/ysharma3501/LinaCodec.git

Recommended for English synthesis (includes phonemizer):

pip install "LuxTTS-mlx[phonemize]" -f https://k2-fsa.github.io/icefall/piper_phonemize.html

From source:

git clone https://github.com/jishnuvenugopal/LuxTTS-mlx.git
cd LuxTTS-mlx
pip install -r requirements.txt

Load model:

from luxtts_mlx import LuxTTS

# load model on GPU
lux_tts = LuxTTS('YatharthS/LuxTTS', device='cuda')

# load model on CPU
# lux_tts = LuxTTS('YatharthS/LuxTTS', device='cpu', threads=2)

# load model on MPS for macs
# lux_tts = LuxTTS('YatharthS/LuxTTS', device='mps')

# load model on MLX (Apple Silicon, Python 3.11+)
# lux_tts = LuxTTS('YatharthS/LuxTTS', device='mlx')

Note: On MLX, both diffusion and vocoder run in MLX by default.

Note: The MLX vocoder path uses vocos-mlx and will download the LuxTTS vocoder weights on first run.

Important: English synthesis requires piper_phonemize. Install with: pip install "LuxTTS-mlx[phonemize]" -f https://k2-fsa.github.io/icefall/piper_phonemize.html

Simple inference

import soundfile as sf
from IPython.display import Audio

text = "Hey, what's up? I'm feeling really great if you ask me honestly!"

## change this to your reference file path, can be wav/mp3
prompt_audio = 'audio_file.wav'

## encode audio(takes 10s to init because of librosa first time)
encoded_prompt = lux_tts.encode_prompt(prompt_audio, rms=0.01)

## generate speech
final_wav = lux_tts.generate_speech(text, encoded_prompt, num_steps=4)

## save audio
final_wav = final_wav.numpy().squeeze()
sf.write('output.wav', final_wav, 48000)

## display speech
if display is not None:
  display(Audio(final_wav, rate=48000))

Inference with sampling params:

import soundfile as sf
from IPython.display import Audio

text = "Hey, what's up? I'm feeling really great if you ask me honestly!"

## change this to your reference file path, can be wav/mp3
prompt_audio = 'audio_file.wav'

rms = 0.01 ## higher makes it sound louder(0.01 or so recommended)
t_shift = 0.9 ## sampling param, higher can sound better but worse WER
num_steps = 4 ## sampling param, higher sounds better but takes longer(3-4 is best for efficiency)
speed = 1.0 ## sampling param, controls speed of audio(lower=slower)
return_smooth = True ## sampling param, smoother/clearer default output path
ref_duration = 5 ## Setting it lower can speedup inference, set to 1000 if you find artifacts.

## encode audio(takes 10s to init because of librosa first time)
encoded_prompt = lux_tts.encode_prompt(prompt_audio, duration=ref_duration, rms=rms)

## generate speech
final_wav = lux_tts.generate_speech(text, encoded_prompt, num_steps=num_steps, t_shift=t_shift, speed=speed, return_smooth=return_smooth)

## save audio
final_wav = final_wav.numpy().squeeze()
sf.write('output.wav', final_wav, 48000)

## display speech
if display is not None:
  display(Audio(final_wav, rate=48000))

Tips

• Please use at minimum a 3 second audio file for voice cloning.
• Best cloning quality comes from clean real speech (single speaker, no music/noise), typically 3-8 seconds.
• return_smooth=True is the default and usually sounds clearer; use False for sharper 48k output.
• Lower t_shift for less possible pronunciation errors but worse quality and vice versa.

CLI quick test (no prompt)

luxtts-mlx "Hello from MLX!" --out output.wav --device mlx

Use --no-return-smooth if you want the sharper 48k path. Defaults are tuned for clarity: --num-steps 5, --speed 0.92, --duration-pad-frames 16. Default output peak normalization is enabled: --output-peak 0.92 (set --output-peak 0 to disable). If output feels too loud on speakers/headphones, use --output-peak 0.8. Prompt preprocessing is enabled by default: silence edge trim (--trim-prompt-silence) and safe RMS clamping (--prompt-rms-min 0.006, --prompt-rms-max 0.03).

CLI with prompt + optional prompt text

luxtts-mlx --text "Hello from MLX!" --prompt /path/to/prompt.wav --out output.wav --device mlx
luxtts-mlx --text "Hello from MLX!" --prompt /path/to/prompt.wav --prompt-text "Hello." --out output.wav --device mlx

Tip: providing --prompt-text skips Whisper prompt transcription load, which is faster and avoids extra multiprocessing warnings. For long/noisy prompt files, use --prompt-start and --ref-duration to target a clean segment. If prompt text is mis-transcribed or repeated, always pass explicit --prompt-text. If trimming is too aggressive for your clip, pass --no-trim-prompt-silence or lower trim sensitivity with --prompt-silence-threshold-db -48.

Optional fallback: torch vocoder with MLX diffusion

luxtts-mlx "Hello from MLX!" --prompt /path/to/prompt.wav --out output.wav --device mlx --vocoder torch

If you hit a Metal kernel error such as Unable to load function four_step_mem_8192..., re-run with --vocoder torch.

Automated recursive feedback loop (Torch vs MLX)

Use this to auto-run multi-round quality checks, compare backends, and tune params until thresholds pass:

./.venv/bin/python tools/feedback_loop.py \
  --text "This is a stability smoke test for LuxTTS MLX prompt preprocessing." \
  --prompt /path/to/prompt1.wav \
  --prompt /path/to/prompt2.wav \
  --prompt-text "Exact transcript of prompt clips." \
  --device mlx \
  --vocoder-set both \
  --max-rounds 4 \
  --max-candidates-per-round 10 \
  --keep-wavs final \
  --round-history none \
  --compact-report \
  --no-write-case-reports \
  --out-dir feedback-loop-runs

Outputs:

• Storage-optimized by default:
  • only final best WAV per prompt (--keep-wavs final)
  • optional zero round history (--round-history none)
  • no per-case JSON unless enabled (--write-case-reports)
• Aggregate summary: feedback-loop-runs/SUMMARY.md.
• Uses ASR similarity + start/tail artifact ratios + repetition ratio to pick winners.
• Includes prompt transcript safeguards for auto-transcribed prompt text:
  • --auto-prompt-text-policy strict (default): fail if transcript looks unstable.
  • --auto-prompt-text-policy target-fallback: use target text when auto transcript looks unstable.
  • --auto-prompt-text-policy hello-fallback: use "Hello." fallback.
• Reuses prompt encodings across candidate variants to speed multi-round loops.

If you need full debug artifacts, use --keep-wavs all --no-compact-report --keep-asr-text.

5-scenario quality/performance matrix (recommended smoke gate)

Run a fixed short/medium/long/numeric/punctuation matrix with storage-safe outputs:

./.venv/bin/python tools/scenario_matrix.py \
  --prompt /path/to/prompt.wav \
  --prompt-text "Exact transcript of prompt clip." \
  --device mlx \
  --vocoder-set both \
  --keep-wavs best \
  --out-dir scenario-matrix-runs

Outputs:

• scenario-matrix-runs/report.json
• scenario-matrix-runs/SUMMARY.md
• one best wav per scenario (or all/none via --keep-wavs)

Info

Q: How is this different from ZipVoice?

A: LuxTTS uses the same architecture but distilled to 4 steps with an improved sampling technique. It also uses a custom 48khz vocoder instead of the default 24khz version.

Q: Can it be even faster?

A: Yes, currently it uses float32. Float16 should be significantly faster(almost 2x).

Roadmap

• Release model and code
• Huggingface spaces demo
• Release MPS support (thanks to @builtbybasit)
• Release code for float16 inference

Acknowledgments

• ZipVoice for their excellent code and model.
• Vocos for their great vocoder.

Final Notes

The model and code are licensed under the Apache-2.0 license. See LICENSE for details.

Stars/Likes would be appreciated, thank you.

Email: yatharthsharma350@gmail.com