LocalKin Service Audio

Local Voice AI Platform - Speech-to-Text and Text-to-Speech with Chinese language support, voice cloning, and Claude integration via MCP.

What's New in v2.1.0

• 50 current models, one name each, on any machine. Qwen3-ASR, Fun-ASR-Nano, FireRedASR2, Parakeet, Nemotron, VibeVoice-ASR; Qwen3-TTS, VoxCPM2, CosyVoice3, IndexTTS-2, Fish Audio S2 Pro, Chatterbox and more — picked by Hugging Face downloads, likes and trending. On a Mac a model runs on MLX ([mlx] extra); on CUDA or CPU it runs in its own environment, built automatically the first time.
• Benchmarked on real speech (FLEURS, Mac Studio M3 Ultra). Qwen3-ASR 1.7B beats SenseVoice-Small in Mandarin (9.4 vs 10.5% CER), Cantonese (6.6 vs 8.7%) and English (3.9 vs 8.2% WER) while running faster; Fun-ASR-Nano has the lowest Mandarin CER (8.6%). Qwen3-TTS 0.6B synthesizes 5× faster than real time and handles Chinese–English code-switching that Kokoro misreads. Full tables in the changelog.
• Every model in kin audio models loads, and the HTTP server serves all of them. /synthesize gains instruct for tone and voice design.
• Fixes found on a clean machine: SenseVoice no longer runs pip install on load, Kokoro can't take the server down when its spaCy setup fails, CosyVoice works for the first time, MusicGen plays at the right pitch.

Recommended:

kin audio serve qwen3-asr:1.7b --port 8000 --emotion sensevoice:small   # STT, with SenseVoice's emotion labels
kin audio serve qwen3-tts:0.6b --port 8001                              # TTS, accepts Kokoro voice ids

See CHANGELOG.md for full history.

Features

• STT: Qwen3-ASR, Fun-ASR-Nano, FireRedASR2, GLM-ASR, Parakeet, Canary, Nemotron, Voxtral Realtime, VibeVoice-ASR, Whisper (openai / faster-whisper / whisper.cpp / MLX), SenseVoice, Paraformer, Moonshine — full list
• TTS: Qwen3-TTS, VoxCPM2, CosyVoice3, IndexTTS-2, Fish Audio S2 Pro, Chatterbox, Spark-TTS, OmniVoice, CSM, Dia, Higgs Audio, Kokoro, ChatTTS, F5-TTS and more
• Any platform: one model name everywhere — MLX on Apple Silicon, torch on CUDA/CPU in auto-built isolated environments
• Music Generation: HeartMuLa (multilingual, tag-based), MusicGen
• Voice Activity Detection: Engine-agnostic /vad endpoint via TEN-VAD; or built-in Silero VAD inside faster-whisper
• Chinese Language Support: Optimized models for Mandarin, Cantonese, and mixed Chinese-English
• Voice Cloning: Zero-shot voice cloning with F5-TTS and CosyVoice
• MCP Integration: Use with Claude Code and Claude Desktop
• WebSocket Streaming: Real-time transcription and synthesis
• REST API: FastAPI-based server with OpenAPI docs; OpenAI-compatible /v1/audio/transcriptions
• Subtitle Output: Direct SRT / WebVTT generation from /transcribe?response_format=srt|vtt

Quick Start

# Install (uv recommended)
uv pip install localkin-service-audio

# Get model recommendations for your hardware
kin audio recommend

# View configuration
kin audio config

# Transcribe audio
kin audio transcribe audio.wav

# Text-to-speech
kin audio tts "Hello world"

# Generate music (with Chinese support!)
kin audio music generate "在月光下弹钢琴"  # Chinese lyrics
kin audio music generate "happy wedding" --tags "piano,romantic,wedding" --model heartmula:3b

# Real-time listening (microphone)
kin audio listen

# Voice AI conversation
kin audio listen --llm ollama --tts --stream

# List available models
kin audio models

# Start API server
kin audio serve --port 8000

# Start web interface
kin web

Installation

Using uv (recommended — 10x faster)

This project has heavy ML dependencies (~4GB: PyTorch, Whisper, transformers). uv resolves and installs them 10-100x faster than pip.

# Install uv (one-time)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install
uv pip install localkin-service-audio

# Or from source
git clone https://github.com/LocalKinAI/localkin-service-audio.git
cd localkin-service-audio
uv sync

Using in a new terminal: The virtual environment needs to be activated each session:

# Option 1: Activate the venv
source .venv/bin/activate
kin audio models

# Option 2: Use uv run (no activation needed)
uv run kin audio models

To auto-activate, add to your ~/.zshrc or ~/.bashrc:

# Activate .venv automatically when entering a project directory
cd() { builtin cd "$@" && [ -f .venv/bin/activate ] && source .venv/bin/activate; }

Using pip

pip install localkin-service-audio

pip works but is significantly slower due to dependency resolution with large ML packages. Expect 10-30 minutes on first install.

Upgrading

# Upgrade to latest version
uv pip install --upgrade localkin-service-audio

# If upgrading from v2.0.3 or earlier, also upgrade torch (required for v2.0.4+)
uv pip install --upgrade torch torchaudio torchvision

Optional Dependencies

# Chinese language models
uv pip install localkin-service-audio[chinese]

# Voice cloning models
uv pip install localkin-service-audio[cloning]

# MCP server for Claude
uv pip install localkin-service-audio[mcp]

# Apple Silicon: run catalog models on MLX (fastest on a Mac). Optional —
# without it they run in auto-built torch environments like everywhere else.
uv pip install "localkin-service-audio[mlx]"

# All features
uv pip install localkin-service-audio[all-new]

Replace uv pip with pip if not using uv.

CLI Usage

Speech-to-Text

# Basic transcription (auto-selects best model)
kin audio transcribe audio.wav

# Specify model
kin audio transcribe audio.wav --model whisper-cpp:base
kin audio transcribe audio.wav --model faster-whisper:large-v3
kin audio transcribe audio.wav --model sensevoice:small  # Chinese

# With language hint
kin audio transcribe audio.wav --language zh

# Output formats
kin audio transcribe audio.wav --format json
kin audio transcribe audio.wav --format srt --timestamps

Text-to-Speech

# Basic synthesis (uses Kokoro with af_heart voice)
kin audio tts "Hello world"

# List all available voices
kin audio tts "" --model kokoro --list-voices

# American English voices
kin audio tts "Hello world" --voice af_bella       # Bella (Female)
kin audio tts "Hello world" --voice am_adam         # Adam (Male)
kin audio tts "Hello world" --voice af_nova         # Nova (Female)

# British English voices
kin audio tts "Good morning" --voice bf_emma        # Emma (British Female)
kin audio tts "Good morning" --voice bm_george      # George (British Male)

# Chinese (Mandarin) voices
kin audio tts "你好世界" --voice zf_xiaoxiao         # Xiaoxiao (Chinese Female)
kin audio tts "今天天气真好" --voice zm_yunyang       # Yunyang (Chinese Male)

# Japanese voices
kin audio tts "こんにちは" --voice jf_alpha           # Alpha (Japanese Female)
kin audio tts "ありがとう" --voice jm_kumo            # Kumo (Japanese Male)

# French, Spanish, Italian, Hindi, Portuguese
kin audio tts "Bonjour le monde" --voice ff_siwis   # French
kin audio tts "Hola mundo" --voice ef_dora           # Spanish
kin audio tts "Ciao mondo" --voice if_sara           # Italian
kin audio tts "नमस्ते" --voice hf_alpha              # Hindi
kin audio tts "Olá mundo" --voice pf_dora            # Portuguese

# Adjust speech speed (0.5 = slow, 2.0 = fast)
kin audio tts "Hello world" --speed 0.8
kin audio tts "Hello world" --speed 1.5

# Save to file
kin audio tts "Hello world" --output speech.wav

# Save without auto-playing
kin audio tts "Hello world" --output speech.wav --no-play

# CosyVoice for Chinese (voice cloning capable)
kin audio tts "你好世界" --model cosyvoice:300m --voice 中文女

Music Generation

# MusicGen — text-to-music (small/medium/large)
kin audio music generate "calm piano melody"
kin audio music generate "upbeat electronic" --duration 20 --model musicgen:medium
kin audio music generate "ambient soundscape" -o ambient.wav --device mps

# HeartMuLa — multilingual with Chinese lyrics support
kin audio music generate "在月光下弹钢琴" --model heartmula:3b
kin audio music generate "happy wedding day" --tags "piano,romantic,wedding" --model heartmula:3b --duration 30
kin audio music generate "春天来了，鸟儿在唱歌" --tags "acoustic,happy,upbeat" -o spring.wav

# List music models and requirements
kin audio music models
kin audio music models --verbose

HeartMuLa style tags: piano, acoustic, electric, synthesizer, happy, sad, romantic, calm, upbeat, wedding, ambient, orchestral, rock, pop, jazz, folk, classical, cinematic

HeartMuLa setup — auto-installs on first use, or pull in advance:

kin audio pull heartmula:3b

Real-time Listening

# Basic real-time transcription
kin audio listen

# With TTS echo
kin audio listen --tts --tts-model kokoro

# Voice AI with LLM (requires Ollama)
kin audio listen --llm ollama --tts --stream

# Custom models
kin audio listen --model sensevoice:small --language zh --tts --tts-model cosyvoice:300m

# Adjust silence detection
kin audio listen --silence-threshold 0.02 --silence-duration 2.0

Model Management

# List all models with availability status
kin audio models

# Filter by type, language, engine, or tag
kin audio models --type stt
kin audio models --type tts
kin audio models --language zh
kin audio models --engine kokoro
kin audio models --tag voice-cloning
kin audio models --search whisper

# Pull a model
kin audio pull whisper-cpp:base
kin audio pull heartmula:3b

# Remove a model
kin audio rm whisper-cpp:base

# Add a model from a template
kin audio add-model --template whisper_stt --name my-whisper

# Add a model from HuggingFace
kin audio add-model --repo openai/whisper-medium --name whisper-med --type stt

# List available model templates
kin audio list-templates

Model Recommendations

# Get hardware-aware model recommendations
kin audio recommend

# With detailed hardware info
kin audio recommend --verbose

The recommend command detects your hardware (GPU, RAM, CPU) and suggests optimal STT/TTS models for your system.

Configuration

# View configuration overview
kin audio config

# Show configuration file paths
kin audio config --path

# Show all registered models
kin audio config --models

# Initialize config directory with sample config
kin audio config --init

# Change settings
kin audio config set default_tts_model kokoro
kin audio config set default_stt_model faster-whisper:large-v3
kin audio config set api_port 9000
kin audio config set default_device cuda

Configuration files are stored in $LOCALKIN_HOME/ (default: ~/.localkin-service-audio/).

Set LOCALKIN_HOME to relocate all data (cache, config, models) to another disk:

export LOCALKIN_HOME="/path/to/large/disk/.localkin-service-audio"

System Status & Diagnostics

# Check system status (libraries, registry, cache)
kin audio status

# Show cache info
kin audio cache info

# Clear cache for a specific model
kin audio cache clear whisper-large

# Clear all cached models
kin audio cache clear

# Show running LocalKin Audio servers
kin audio ps

API Server

# Start REST API server
kin audio serve --port 8000

# Start web interface
kin web --port 5000

Supported Models

kin audio models shows all 76 models with real-time availability status:

• ✅ Ready — engine installed, usable now
• 📦 Not installed — strategy code exists, just needs pip install

Current models (catalog)

Use the same name on any machine. Mac (MLX) needs Apple Silicon and the [mlx] extra; torch runs on CUDA, CPU or MPS in an environment built on first use (needs uv; set LOCALKIN_AUTO_INSTALL=0 to refuse, and kin audio pull <model> to build ahead of time). Weights download on first use. Repos and parameters live in catalog.py.

Speech-to-Text

Text-to-Speech

The torch backends follow each project's documented API. Qwen3-ASR, Qwen3-TTS (MPS) and CosyVoice3 (CPU) have run end to end on the torch path; CUDA runs are awaiting reports — issues welcome. Benchmark results cover 8 STT and 9 TTS models.

Built-in engines

Speech-to-Text

Tip — Voice Activity Detection has two paths since v2.0.12:

1. Inline with transcription — use a faster-whisper:* model and pass enable_vad=true to /transcribe. Silero VAD is bundled inside the engine; transitions are merged into the resulting transcript.
2. Standalone, engine-agnostic — call POST /vad (always available, no model required). Backed by TEN-VAD — 731 KB native macOS arm64 binary, faster transitions than Silero. Returns raw speech segments so you can chunk audio before transcription or use it for diarization-lite workflows. See the /vad endpoint docs.

Text-to-Speech

Music Models

Kokoro Voice Reference

Kokoro supports 54 voices across 9 languages. Voice IDs follow the pattern {lang}{gender}_{name}:

Python API

from localkin_service_audio import AudioEngine, transcribe, synthesize

# Quick functions
result = transcribe("audio.wav", model="whisper-cpp:base")
print(result.text)

audio = synthesize("Hello world", model="kokoro")
audio.save("output.wav")

# Full engine control
engine = AudioEngine()

# Load and use STT
engine.load_stt("whisper-cpp:base")
result = engine.transcribe("audio.wav", language="en")
print(f"Text: {result.text}")
print(f"Language: {result.language}")

# Load and use TTS - English
engine.load_tts("kokoro")
audio = engine.synthesize("Hello world", voice="af_heart")
audio.save("english.wav")

# TTS - Chinese (auto-selects Chinese pipeline)
audio = engine.synthesize("你好世界", voice="zf_xiaoxiao")
audio.save("chinese.wav")

# TTS - Japanese
audio = engine.synthesize("こんにちは世界", voice="jf_alpha")
audio.save("japanese.wav")

# TTS - with speed control
audio = engine.synthesize("Hello", voice="am_adam", speed=0.8)
audio.save("slow.wav")

# List available voices
voices = engine.list_voices()
for v in voices:
    print(f"{v.id}: {v.name} ({v.language}, {v.gender})")

# Voice cloning (with supported models)
engine.load_tts("f5-tts")
audio = engine.clone_voice(
    reference_audio="reference.wav",
    text="Text to speak in cloned voice"
)

MCP Integration

Use LocalKin Audio with Claude Code or Claude Desktop:

# Start MCP server
kin mcp

Add to Claude Desktop config (~/.config/claude/claude_desktop_config.json):

{
  "mcpServers": {
    "localkin-audio": {
      "command": "kin",
      "args": ["mcp"]
    }
  }
}

Available MCP tools:

• transcribe_audio - Transcribe audio files
• synthesize_speech - Generate speech from text
• clone_voice - Clone voice from reference audio
• list_models - List available models
• list_voices - List available voices

REST API

Start the server:

kin audio serve --port 8000

Endpoints

POST /transcribe - Transcribe audio

# Basic (language is a query parameter; omit it to auto-detect)
curl -X POST "http://localhost:8000/transcribe?language=en" \
  -F "file=@audio.wav"

With SenseVoice as the model, or any model served with --emotion sensevoice:small, the JSON also carries emotion (happy / sad / angry / neutral / …) and audio_events (laughter, applause, …). Treat the label as a hint.

Optional query parameters (added in v2.0.11):

# Markdown transcript with timestamps
curl -X POST 'http://localhost:8000/transcribe?response_format=markdown' \
  -F 'file=@meeting.wav'

# SRT subtitles for video captioning
curl -X POST 'http://localhost:8000/transcribe?response_format=srt' \
  -F 'file=@video.wav' > captions.srt

# Low-VRAM long-audio: VAD + smaller chunks
curl -X POST 'http://localhost:8000/transcribe?chunk_length_s=15&enable_vad=true' \
  -F 'file=@long.wav'

# JSON with segment timestamps (no shape change to existing callers
# unless you opt in with timestamps=true)
curl -X POST 'http://localhost:8000/transcribe?timestamps=true' \
  -F 'file=@audio.wav'

POST /synthesize - Synthesize speech (returns audio/wav)

curl -X POST "http://localhost:8001/synthesize" \
  -H "Content-Type: application/json" \
  -d '{"text": "今天天气很好", "speaker": "vivian", "instruct": "用开心的语气"}' \
  --output speech.wav

GET /voices - Voices of the TTS model

curl "http://localhost:8001/voices"
# {"model": "qwen3-tts:0.6b", "multilingual": true, "default_voice": "vivian",
#  "voices": [{"id": "vivian", "name": "Vivian · 明亮女声", "language": "zh", "gender": "female"}, ...]}

multilingual: true means every voice reads every language the model supports, so send mixed-language text in one request rather than splitting it between voices (which single-language Kokoro voices need).

POST /vad - Detect speech segments (no transcription)

Engine-agnostic Voice Activity Detection via TEN-VAD (731 KB native macOS arm64 binary, ~0.016 RTF on M1, ~100-300 ms faster transitions than Silero). Useful for chunking long audio before transcription, or for VAD-only workflows.

# Install the optional VAD extra first
pip install 'localkin-service-audio[vad]'

curl -X POST "http://localhost:8000/vad" \
  -F "file=@meeting.wav"

# Output:
# {
#   "backend": "ten-vad",
#   "duration": 132.4,
#   "speech_segments": [
#     {"start": 1.2, "end": 5.8, "duration": 4.6},
#     ...
#   ],
#   "total_speech_duration": 48.3
# }

Tunable parameters (all optional query strings):

GET /models - List models

curl "http://localhost:8000/models"

GET /health - Readiness. 503 with the missing package when the model's backend isn't installed; loaded: true once the model is in memory (serve preloads by default).

WebSocket /stream - Real-time transcription

const ws = new WebSocket("ws://localhost:8000/stream");
ws.send(audioChunk);
ws.onmessage = (e) => console.log(JSON.parse(e.data).text);

Configuration

Environment Variables

# Base directory for all data (cache, config, models)
export LOCALKIN_HOME="/Volumes/Data/.localkin-service-audio"

# Override individual directories
export LOCALKIN_CACHE_DIR="/tmp/my-cache"
export LOCALKIN_CONFIG_DIR="/path/to/config"
export LOCALKIN_MODELS_DIR="/path/to/models"

# Default engine settings
export LOCALKIN_DEFAULT_STT="faster-whisper:large-v3"
export LOCALKIN_DEFAULT_TTS="kokoro"
export LOCALKIN_DEVICE=cuda  # or cpu, mps, auto

# API server
export LOCALKIN_API_HOST="127.0.0.1"
export LOCALKIN_API_PORT="8000"

Custom Models

Create $LOCALKIN_HOME/models.json (default: ~/.localkin-service-audio/models.json):

{
  "models": {
    "my-custom-model": {
      "type": "stt",
      "engine": "whisper",
      "model_size": "base",
      "languages": ["en", "zh"],
      "description": "My custom model"
    }
  }
}

Architecture

LocalKin Audio v2.0 uses a modular architecture:

• Strategy Pattern: Pluggable STT/TTS engines
• Facade Pattern: AudioEngine provides unified interface
• Registry Pattern: Centralized model configuration
• Singleton Pattern: Shared engine instance

localkin_service_audio/
├── core/
│   ├── audio_processing/
│   │   ├── engine.py          # AudioEngine facade
│   │   ├── stt/               # STT strategies
│   │   │   ├── base.py
│   │   │   ├── whisper_strategy.py
│   │   │   ├── sensevoice_strategy.py
│   │   │   └── ...
│   │   └── tts/               # TTS strategies
│   │       ├── base.py
│   │       ├── kokoro_strategy.py
│   │       ├── cosyvoice_strategy.py
│   │       └── ...
│   ├── config/
│   │   └── model_registry.py  # Model registry
│   └── types.py               # Core dataclasses
├── cli/                       # Click CLI
├── api/                       # FastAPI server
├── mcp/                       # MCP server
└── ui/                        # Web interface

Development

# Clone repository
git clone https://github.com/LocalKinAI/localkin-service-audio.git
cd localkin-service-audio

# Install with dev dependencies (uv recommended)
uv venv && source .venv/bin/activate
uv pip install -e ".[dev]"

# Or with pip
pip install -e ".[dev]"

# Run tests
pytest tests/

# Run linting
ruff check .
black --check .

Tip: With uv, you can skip activation and run commands directly:

uv run kin audio models
uv run pytest tests/

Troubleshooting

Model Loading Errors

# Check model is registered
kin audio models

# Pull the model
kin audio pull whisper-cpp:base

# Check system info
kin info --verbose

PyTorch Version

Requires torch >= 2.6.0. Older versions will fail to load models that only ship .bin weights (e.g. MusicGen medium/large) due to a torch.load security check (CVE-2025-32434).

# Check your version
python -c "import torch; print(torch.__version__)"

# Upgrade if needed (keep torchvision in sync)
pip install "torch>=2.6.0" "torchaudio>=2.6.0" "torchvision>=0.21"

macOS 27: __thread_bss / scipy import error

scipy 1.15.3, the last release with Python 3.10 wheels, fails to load on macOS 27. Create the environment with Python 3.11+:

uv venv -p 3.11 && uv pip install -e ".[sensevoice]"

numpy/pandas Binary Incompatibility

If you see numpy.dtype size changed, may indicate binary incompatibility, pandas or scikit-learn was compiled against a different numpy version:

# Fix: force-reinstall the affected packages
uv pip install --force-reinstall numpy pandas scikit-learn

# Or nuke and rebuild the venv
rm -rf .venv && uv venv && uv pip install localkin-service-audio

CUDA/GPU Issues

# Force CPU
kin audio transcribe audio.wav --device cpu

# Check PyTorch CUDA
python -c "import torch; print(torch.cuda.is_available())"

HeartMuLa on Apple Silicon (MPS)

HeartMuLa 3B requires ~12-14GB. On a 16GB Mac, close memory-heavy apps before running. The codec runs on CPU automatically (shared unified memory, no performance impact).

# If you hit OOM, try shorter duration
kin audio music generate "prompt" --model heartmula:3b --duration 5

# Or force CPU (slower but more stable memory management)
kin audio music generate "prompt" --model heartmula:3b --device cpu

Chinese Model Dependencies

SenseVoice and Paraformer run on FunASR, which is an optional extra — it pulls in modelscope and a sizeable tree, so a Whisper-only install shouldn't have to download it.

# Install the extra (declared in pyproject.toml)
uv pip install --project . -e ".[sensevoice]"

# Or plain pip
pip install funasr modelscope

# Then use Chinese models
kin audio transcribe audio.wav --model sensevoice:small

# Or serve them
kin audio serve sensevoice:small --port 8000

Without the extra, the server still starts and /health still reports {"status": "healthy"} — the failure only shows up on the first /transcribe, as funasr not installed. The model weights are downloaded separately and being present is not enough on its own.

License

MIT License - see LICENSE file.

Acknowledgments

• OpenAI Whisper
• faster-whisper
• whisper.cpp
• FunASR
• Kokoro TTS
• CosyVoice
• MusicGen
• HeartMuLa