smoltts_mlx 0.1.0

MLX inference for autoregressive speech models

smolltts-mlx

Installation

Requires working Python instance and Apple Silicon Mac.

pip install smoltts-mlx

Server

Startup

From the CLI, run:

smoltts-server

Options:

• --port (optional): Port to listen on (default: 8000)
• --config (optional): Point to a JSON file. (See below for spec)

Supported voices

As of February 2025, we support these voices from Kokoro:

• American: heart (default), bella, nova, sky, sarah, michael, fenrir, liam
• British: emma, isabella, fable

Voice cloning is currently not supported, but coming soon!

Unfortunately, GitHub doesn't support audio previews, but check out docs/examples for samples.

ElevenLabs endpoints

We support the following two ElevenLabs endpoints (more to come):

• /v1/text-to-speech/$ID with MP3, WAV, and PCM output
• /v1/text-to-speech/$ID/stream (PCM-only for now)

Here's an example with the Python SDK:

from elevenlabs.client import ElevenLabs

client = ElevenLabs(
    # or wherever you're running this on
    base_url="http://localhost:8000",
)

request_gen = client.text_to_speech.convert(
    voice_id="0",
    output_format="mp3_44100_128",
    text="You can turn on latency optimizations at some cost of quality. The best possible final latency varies by model.",
)

OpenAI endpoints

We support /v1/audio/speech (MP3 and WAV).

Here's an example with the OpenAI Python SDK:

from pathlib import Path
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000"
)

speech_file_path = Path(__file__).parent / "speech.mp3"
response = client.audio.speech.create(
    model="tts-1",
    voice="alloy",
    input="Today is a wonderful day to build something people love!",
)
response.stream_to_file(speech_file_path)

Configuration

Default settings are stored by default at ~/Library/Cache/smoltts.

You can also specify a JSON file with --config.

{
  // "checkpoint_dir": "../inits/foobar/"
  "model_id": "jkeisling/smoltts_v0",
  "generation": {
    "default_temp": 0.0,
    "default_fast_temp": 0.5,
    "min_p": 0.1
  },
  "model_type": {
    "family": "dual_ar",
    "codec": "mimi",
    "version": null
  }
}

Library

Basic Usage

from smoltts_mlx import SmolTTS
from IPython.display import Audio

# Initialize model (downloads weights automatically)
model = SmolTTS()

# Basic generation to numpy PCM array
pcm = model("Hello world!")
Audio(pcm, rate=model.sampling_rate)

# Streaming generation for real-time audio
for pcm_chunk in model.stream("This is a longer piece of text to stream."):
    # Yields 80ms PCM frames as they're generated
    process_audio(pcm_chunk)

Voice Selection

# Use a specific voice
pcm = model("Hello!", voice="af_bella")

# Create a custom voice from reference audio
speaker_prompt = model.create_speaker(
    system_prompt="<|speaker:0|>",
    samples=[{
        "text": "This is a sample sentence.",
        "audio": reference_audio  # Numpy array of PCM data
    }]
)

# Generate with custom voice
pcm = model(
    "Using a custom voice created from reference audio.",
    speaker_prompt=speaker_prompt
)

Working with Audio

The model works with raw PCM audio at 24kHz sample rate. For format conversion:

import soundfile as sf

# Save to WAV
sf.write("output.wav", pcm, model.sampling_rate)

# Load reference audio
audio, sr = sf.read("reference.wav")
if sr != model.sampling_rate:
    # Resample if needed
    audio = resampy.resample(audio, sr, model.sampling_rate)

Advanced Configuration

# Use custom model weights
model = SmolTTS(
    model_id="path/to/custom/model",
    checkpoint_dir="/path/to/local/weights"
)

# Access underlying components
mimi_codes = model.codec.encode(pcm)  # Work with Mimi tokens directly

Performance Notes

• First generation may be slower due to model loading and warmup
• Use streaming for longer texts to begin playback before full generation

Requirements

• Apple Silicon Mac (M1/M2/M3)
• Python 3.9 or later

Developing locally

Please use uv.

From the root of this repo:

uv sync --all-packages