saltts 0.1.0

SalTTS - Real-time, controllable, adaptive neural TTS system for AI VTubers

SalTTS

Real-time, controllable, adaptive neural TTS system for AI VTubers

SalTTS is a production-ready text-to-speech system designed for real-time AI VTuber applications with <300ms latency.

Features

• Dual Generator Architecture: Neural (Flow/Diffusion/GAN) + Parametric (WORLD vocoder)
• Real-time Streaming: <300ms end-to-end latency with KV caching
• Adaptive Mixing: RL-based (Soft Actor-Critic) with HNSW memory
• Multi-modal Fusion: 3-channel NMDB integration
• High Quality: Target MOS 4.0+, character consistency 0.85+

Architecture

NMDB (3 channels) → Multi-Modal Fusion → Prosody Generation →
Dual Generators (A+B) → RL Adaptive Mixing → Post-Processing → Audio

Installation

Using Poetry (Recommended)

poetry install

Using pip

pip install -e .

Quick Start

from saltts.inference import SalTTSEngine

# Initialize engine
engine = SalTTSEngine(config_dir="config")

# Generate speech
audio = engine.synthesize(
    text="Hello, this is SalTTS!",
    emotion="neutral",
    speaker_id=0
)

# Save audio
engine.save_audio(audio, "output.wav")

Configuration

Configuration files are located in config/:

• model.yaml - Model architectures
• runtime.yaml - Runtime settings
• training.yaml - Training hyperparameters
• nmdb_channels.yaml - NMDB configuration
• deployment.yaml - Deployment settings

Training

# Stage 1: Train generators
python scripts/train_stage1.py

# Stage 2: Train transformer
python scripts/train_stage2.py

# Stage 3: Train integration
python scripts/train_stage3.py

# Stage 4: RL training
python scripts/train_stage4_rl.py

Performance

• End-to-end latency: <300ms (target <200ms)
• Real-Time Factor: <0.5
• MOS score: 4.0+
• Character consistency: 0.85+

License

MIT License - see LICENSE file for details