modern-tts 0.3.3

A unified, extensible, and modern Python toolkit for LLM-based Text-to-Speech (TTS) synthesis.

Modern TTS

A unified, extensible, and future-proof Python toolkit for locally running state-of-the-art LLM-based Text-to-Speech (TTS) synthesis models.

---

✨ Features

• 🧩 25+ Models — MeloTTS, ChatTTS, CosyVoice, Fish Speech, Parler-TTS, XTTS, GPT-SoVITS, F5-TTS, Qwen3-TTS, GLM-TTS, Index-TTS, MaskGCT, and more
• 🔌 Plugin Architecture — Add new models with @register_model decorator
• 🚀 Hot-Swap — Switch models at runtime without restarting
• 🌍 Multi-Language — Chinese, English, Japanese, Korean, and more
• 🎯 Multi-Task — Speech synthesis, voice cloning, emotion control, style transfer, streaming
• 💻 Local-First — All inference on-device. No APIs. No data leaves your machine.
• 🐍 Modern Python — uv-native packaging, Pydantic configs, rich CLI
• 📦 Zero-Config Auto-Download — GLM-TTS, Index-TTS, RedFire-TTS, CosyVoice, and Fish Speech automatically download repos, dependencies, and weights on first use

---

📦 Installation

# Clone the repository
git clone https://github.com/vra/modern-tts.git
cd modern-tts

# Sync all dependencies (recommended)
uv sync --all-extras

# Or install specific extras only
uv sync --extra melotts --extra chattts --extra glm --extra index

# Or just core dependencies
uv sync

Python 3.10+ recommended. Some models (e.g. Index-TTS) require specific PyTorch / transformers versions—see per-model notes below.

---

🚀 Quick Start

from modern_tts import TTSPipeline

# Synthesize with MeloTTS
pipe = TTSPipeline("melotts-zh")
result = pipe("你好世界，这是语音合成测试。")
result.save("output.wav")

# Switch to ChatTTS for emotional speech
pipe.switch_model("chattts")
result = pipe("这是一个带有情感的语音合成。")
result.save("output_emotion.wav")

# Voice cloning with CosyVoice
pipe.switch_model("cosyvoice-300m")
result = pipe("这是克隆的声音。", task="clone", reference_audio="reference.wav")
result.save("cloned.wav")

# Zero-config voice cloning with GLM-TTS (auto-downloads code)
pipe.switch_model("glm-tts")
result = pipe("你好，这是 GLM-TTS 的语音克隆测试。", task="clone", reference_audio="ref.wav")
result.save("glm_cloned.wav")

# Zero-config voice cloning with Index-TTS (auto-downloads code)
pipe.switch_model("index-tts")
result = pipe("你好，这是 Index-TTS 的语音克隆测试。", task="clone", reference_audio="ref.wav")
result.save("index_cloned.wav")

# Zero-config voice cloning with RedFire-TTS (auto-downloads everything)
pipe.switch_model("redfire-tts")
result = pipe("你好，这是 RedFire-TTS 的语音克隆测试。", task="clone", reference_audio="ref.wav")
result.save("redfire_cloned.wav")

---

🎙️ Supported Models

✅ Ready to use (loadable out-of-the-box)

Model ID	Type	Languages	Modes	Install Extra	Notes
melotts-zh	TTS	zh, en	speak, emotion	--extra melotts	Many text-processing deps (pypinyin, jieba, etc.)
melotts-en	TTS	zh, en	speak, emotion	--extra melotts	English variant
chattts	TTS	zh, en	speak, clone, emotion	--extra chattts	Emotional prosody control
f5-tts	ZS-VC	zh, en, ja, ko	speak, clone, emotion	--extra f5	Requires reference audio for synthesis
glm-tts	ZS-VC	zh, en	speak, clone	--extra glm	Auto-downloads official repo. Heavy deps (transformers, onnxruntime, peft).
index-tts	ZS-VC	zh, en, ja, ko, yue	speak, clone, emotion, style	--extra index	Auto-downloads official repo. Requires Python ≥ 3.10.
moss-tts	TTS	zh, en, ja, ko	speak, emotion	--extra moss	MOSS-TTS-Nano (0.1B), CPU-friendly
piper-tts	TTS	15+	speak	--extra piper	ONNX-based, edge-optimized
redfire-tts	ZS-VC	zh, en	speak, clone	--extra redfire	Auto-downloads repo + weights + all deps (fairseq, etc.)
qwen3-tts-0.6b	ZS-VC	11+	speak, clone	--extra qwen3-tts	Requires qwen-tts package
qwen3-tts-1.7b	ZS-VC	11+	speak, clone	--extra qwen3-tts	Larger Qwen3-TTS variant
xtts-v1	ZS-VC	13+	speak, clone	--extra xtts	Requires coqui-tts
xtts-v2	ZS-VC	13+	speak, clone	--extra xtts	Adds Chinese support
xtts-v2.1	ZS-VC	13+	speak, clone, streaming	--extra xtts	Adds streaming mode
cosyvoice-300m	ZS-VC	zh, en, yue, ja, ko	speak, clone	--extra cosyvoice	Auto-downloads repo + HF weights
cosyvoice-300m-sft	ZS-VC	zh, en, yue, ja, ko	speak, clone, emotion, style	--extra cosyvoice	Auto-downloads, has built-in speaker voices
cosyvoice-300m-instruct	ZS-VC	zh, en, yue, ja, ko	speak, clone, emotion, style	--extra cosyvoice	Auto-downloads, instruction-based control
fishspeech-1.5	ZS-VC	zh, en, ja, ko	speak, clone, emotion	--extra fishspeech	Auto-downloads repo + HF weights

ZS-VC = Zero-Shot Voice Cloning (requires a reference_audio sample).

⚠️ Auto-downloads repo, but requires user-trained weights

These models auto-clone their official repos and auto-install dependencies, but require user-trained model weights for specific voices. They are voice cloning training frameworks — you must train a voice model first.

Model ID	Type	Languages	Modes	Install Extra	Setup Notes
gptsovits	ZS-VC	zh, en, ja, yue	speak, clone	--extra gptsovits	Auto-downloads repo; needs user-trained GPT + SoVITS weights
bertvits2-zh	TTS	zh, en	speak, emotion	--extra bertvits2	Auto-downloads repo; needs user-trained weights. Unmaintained.
bertvits2-en	TTS	en	speak, emotion	--extra bertvits2	Same as above
bertvits2-jp	TTS	ja, en	speak, emotion	--extra bertvits2	Same as above

❌ Temporarily unavailable

Model ID	Reason
maskgct	Custom tokenizer incompatible with generic TextToAudioLLMModel loader
parler-tts-mini	parler-tts package incompatible with transformers >= 4.50
parler-tts-large	Same compatibility issue as parler-tts-mini
pocket-tts	No public repository or weights found (reserved for future implementation)

---

📋 Changelog & API Changes

Latest

New Models

• glm-tts — LLM + Flow Matching zero-shot TTS (Zhipu AI). Merged previous glm-tts-nano-2512 and glm-tts-2512 into a single glm-tts model ID.
• index-tts — Industrial-level multilingual zero-shot voice cloning (IndexTeam).
• redfire-tts — High-quality Chinese/English zero-shot voice cloning (Xiaohongshu/FireRedTeam). Fully auto-download with Python 3.12 + transformers 5.x compatibility.

Zero-Config Auto-Download

• GLM-TTS, Index-TTS, RedFire-TTS, CosyVoice, and Fish Speech no longer require manual environment variables or PYTHONPATH manipulation.
• On first use, the framework automatically:
  1. Clones the official repository to ~/.cache/modern-tts/repos/
  2. Installs all required pip dependencies (including fairseq with Python 3.12 fixes)
  3. Downloads model weights from HuggingFace Hub
  4. Patches upstream code for compatibility (transformers 5.x, WeTextProcessing)
  5. Proceeds with model loading
• You can still override paths via config.extra or environment variables (GLM_TTS_REPO_PATH, INDEX_TTS_REPO_PATH, REDFIRE_TTS_REPO_PATH).

New Infrastructure Modules

• modern_tts.core.hf_hub — HuggingFace Hub download helpers (download_hf_model, get_hf_model_path) so custom-code adapters don't re-implement caching logic.
• modern_tts.core.repo_manager — Generic git repository auto-downloader (ensure_repo, inject_repo_path) used by adapters that depend on upstream code not on PyPI.
• modern_tts.utils.auto_install — Runtime pip dependency installer (ensure_packages, _pip_install) with uv support and pip-name-to-import-name mapping.

Base Class Improvements

• TextToAudioLLMModel.load() now raises a clear NotImplementedError when a subclass has not set PROCESSOR_CLS / MODEL_CLS, signaling that the subclass must override load() for custom loading logic.

Model ID Changes

Old ID	New ID	Note
glm-tts-nano-2512	glm-tts	Merged into unified glm-tts
glm-tts-2512	glm-tts	Merged into unified glm-tts

---

🏗️ Architecture

Modern TTS is built on three layers:

1. TTSPipeline — Unified user API. Handles text normalization, task dispatch, model lifecycle.
2. TTSModel / TextToAudioLLMModel — Adapter layer. New models often need only 8 lines of config via TextToAudioLLMModel.
3. Backends — Transformers, vLLM, ONNX Runtime.

Adding a New Model

from modern_tts.core.audio_llm import TextToAudioLLMModel
from modern_tts.core.registry import register_model

@register_model("my-tts-1b")
class MyTTS1B(TextToAudioLLMModel):
    HF_PATH = "org/MyTTS-1B"
    PROCESSOR_CLS = "transformers.AutoTokenizer"
    MODEL_CLS = "transformers.AutoModelForTextToWaveform"
    SUPPORTED_LANGUAGES = {"zh", "en"}
    DEFAULT_SAMPLE_RATE = 24000

    @property
    def model_id(self) -> str:
        return "my-tts-1b"

That's it. The registry auto-discovers it at runtime.

---

🤝 Contributing

See Contributing Guide for development setup, code style, and PR checklist.

---

📄 License

Apache-2.0