smart-tts

High-level Python library for expressive speech production: Fish Audio TTS, ElevenLabs music and ambient beds, and ffmpeg layer mixing.

Pass raw text (optionally with SSML <break> tags) — the library converts pauses to Fish Audio paralanguage, synthesizes speech in one continuous pass, and can mix music and ambient underneath.

Features

• SmartTTS facade — one pipeline from text to audio
• Fish Audio speech — single-pass synthesis via s2.1-pro (fallback to s2.1-pro-free on 402)
• SSML breaks → Fish S2 tags — <break time="1.2s"/> becomes [long pause]
• ElevenLabs beds — optional music (music.compose) and ambient (text_to_sound_effects)
• ffmpeg mixing — speech + music + ambient with volume weights
• Sync & async API — SmartTTS / AsyncSmartTTS with the same signatures
• Voice registry — local diskcache for registered Fish reference_id voices

Requirements

• Python 3.11+
• ffmpeg and ffprobe in PATH (only when mixing layers)

Installation

pip install smart-tts

Or from source:

git clone https://github.com/vpuhoff/smart-tts.git
cd smart-tts
uv sync --dev

Quick start

1. Copy .env.example to .env and set your API keys:

cp .env.example .env

2. Run synthesis:

from pathlib import Path

from smart_tts import SmartTTS, SynthesisTask, VoiceSettings

tts = SmartTTS.from_env()
tts.sync_voices()

result = tts.synthesize_to_file(
    SynthesisTask(
        text='Центр, <break time="1.2s" /> на связи резидентура.',
        language="ru",
        style="serious",
        emotion="serious",
        voice_id="67d37d81cb7340b391e9461d6671de03",
        voice_settings=VoiceSettings(temperature=0.7, speed=1.0),
    ),
    Path("output.mp3"),
)

print(result.enhanced_text)

See example.py for a full demo: detective radio report with speech variants, custom music, and remix.

uv run python example.py
uv run python example.py --variants 2
uv run python example.py --remix-only --music back.mp3

Synthesis with music and ambient

Pass bed prompts or file paths in SynthesisTask — synthesize() generates speech, then mixes layers automatically:

result = tts.synthesize(
    SynthesisTask(
        text="...",
        voice_id="your-fish-reference-id",
        music_prompt="Melancholic noir piano, instrumental, no vocals",
        ambient_prompt="Subtle radio hum, tape hiss, seamless loop",
        music_volume=0.32,
        ambient_volume=0.18,
        bed_weight=0.68,
    )
)

Or provide pre-recorded files:

SynthesisTask(
    text="...",
    music_path="back.mp3",
    ambient_path="ambient.wav",
)

ELEVENLABS_API_KEY is required for API-generated beds. Custom files work without it.

Task parameters

Core fields

Mixing fields

SSML breaks

# Input
'Срочное донесение. <break time="1.2s" /> Обнаружена цель.'

# After enhance_text (Fish S2 [bracket] tags)
'Срочное донесение. [long pause] Обнаружена цель.'

Emotion tags

Fish Audio S2/S2.1 interprets [bracket] tags as delivery hints (not spoken text). Parenthesis prose like (soft tone) is not supported and may be read aloud.

Configuration

Required

Optional

Programmatic configuration:

from smart_tts import SmartTTS, SmartTTSConfig, TTSModel

config = SmartTTSConfig(
    fish_api_key="...",
    elevenlabs_api_key="...",  # optional
    default_model=TTSModel.ELEVEN_V3,
    default_voice_id="67d37d81cb7340b391e9461d6671de03",
)
tts = SmartTTS(config)

Usage

Synthesis pipeline

from smart_tts import SmartTTS, SynthesisTask, TTSModel, VoiceSettings

with SmartTTS.from_env() as tts:
    tts.sync_voices()
    result = tts.synthesize(
        SynthesisTask(
            text="Срочное донесение.",
            voice_id="67d37d81cb7340b391e9461d6671de03",
            model=TTSModel.ELEVEN_V3,
            emotion="serious",
            voice_settings=VoiceSettings(temperature=0.7),
        )
    )
    audio_bytes = result.audio
    prepared_text = result.enhanced_text

Generation templates

Use GenerationTemplate to bundle speech, background, and mix settings. Pass only the script text at synthesis time:

from pathlib import Path

from smart_tts import INVESTIGATION, GenerationTemplate, SmartTTS, synthesize_with_template

# Built-in preset
with SmartTTS.from_env() as tts:
    speech = tts.synthesize_text(
        'Срочное донесение. <break time="1.2s" /> Обнаружена цель.',
        INVESTIGATION,
        mix=False,  # speech only
    )
    tts.synthesize_text_to_file(
        "Конец связи.",
        INVESTIGATION,
        Path("output/speech.mp3"),
        mix=False,
    )
    tts.remix_file(
        Path("output/speech.mp3"),
        Path("output/final.mp3"),
        INVESTIGATION,
    )

# Custom template or overrides
template = INVESTIGATION.with_overrides(
    speech_volume=1.2,
    music_path="back.mp3",
    ambient_path=None,
)
result = synthesize_with_template("Привет!", template, mix=True)

# Load/save JSON recipes (see templates/investigation.json)
template = GenerationTemplate.from_json_file("templates/investigation.json")

Preview prepared text without TTS

prepared = tts.enhance_text_only(
    SynthesisTask(
        text='Центр, <break time="1.2s" /> на связи.',
        emotion="warm",
    )
)

One-liner

from smart_tts import synthesize

result = synthesize(
    "Привет, мир!",
    language="ru",
    style="neutral",
)

Async API

import asyncio
from pathlib import Path

from smart_tts import AsyncSmartTTS, SynthesisTask, asynthesize

async def main() -> None:
    async with AsyncSmartTTS.from_env() as tts:
        result = await tts.synthesize_to_file(
            SynthesisTask(text="Привет!", language="ru"),
            Path("output.mp3"),
        )
        print(result.enhanced_text)

asyncio.run(main())

Pydantic AI tools (optional)

Install the extension::

pip install smart-tts[pydantic-ai]

Register a toolset with an agent and pass SmartTTSDeps (or any deps implementing HasSmartTTS) as runtime dependencies:

from pathlib import Path

from pydantic_ai import Agent

from smart_tts.async_tts import AsyncSmartTTS
from smart_tts.extensions.pydantic_ai import (
    SmartTTSDeps,
    create_smart_tts_toolset,
    resolve_openrouter_model,
)
from smart_tts.templates import default_template_registry

async with AsyncSmartTTS.from_env() as tts:
    agent = Agent(
        resolve_openrouter_model(),  # OPENROUTER_API_KEY + model from .env
        deps_type=SmartTTSDeps,
        toolsets=[create_smart_tts_toolset(registry=default_template_registry())],
    )
    result = await agent.run(
        "Озвучь донесение: Обнаружена цель.",
        deps=SmartTTSDeps(tts=tts, output_dir=Path("output")),
    )

For custom template sources (e.g. PostgreSQL) and existing AgentDeps, see agents.md — Integrating with AgentDeps.

Tools:

Полный пример: example_pydantic_ai.py

pip install smart-tts[pydantic-ai]
uv run python example_pydantic_ai.py              # прямой вызов tools
uv run python example_pydantic_ai.py --agent        # агент + TestModel
uv run python example_pydantic_ai.py --agent --live # агент + OpenRouter LLM

Requires OPENROUTER_API_KEY. Model is taken from --model, PYDANTIC_AI_MODEL, or OPENROUTER_API_TTS_PROMPT_MODEL.

OpenTelemetry tracing (optional)

pip install smart-tts[otel-sdk]

from smart_tts.extensions.otel import configure_tracing, shutdown_tracing

configure_tracing()  # uses OTEL_EXPORTER_OTLP_ENDPOINT
with SmartTTS.from_env() as tts:
    tts.synthesize_text("Привет!", INVESTIGATION, mix=False)
shutdown_tracing()

Spans are emitted for synthesis, Fish/ElevenLabs calls, ffmpeg mix, and Pydantic AI tools. Without opentelemetry-api, tracing is a no-op.

Voice registry

voices = tts.list_voices()
voice = tts.get_voice("reference-id")
tts.sync_voices(force=True)  # refresh default voice in cache

Fish voices are referenced by reference_id from the Fish Audio console. Register custom voices via VoiceRegistry.register_voice() or set FISH_DEFAULT_VOICE_ID.

Models (TTSModel)

Legacy enum names map to Fish Audio models:

Package layout

smart_tts/
├── tts.py, async_tts.py     # SmartTTS facade
├── telemetry.py             # OpenTelemetry spans (optional)
├── templates.py             # GenerationTemplate presets
├── config.py, models.py
├── client/
│   ├── fish.py              # Fish Audio TTS
│   └── elevenlabs.py        # Music + ambient beds
├── script/breaks.py         # SSML → Fish paralanguage
├── audio/mixer.py           # ffmpeg mix_tracks
├── text.py                  # prepare_text()
└── voices/registry.py       # diskcache voice registry

Development

uv sync --dev
uv run pytest
uv run ruff check .

License

MIT — see LICENSE.

Links

• Agent integration guide
• GitHub repository
• PyPI package
• Design specification (Russian)