LocalKin Service Audio

Local Voice AI Platform - Speech-to-Text and Text-to-Speech with Chinese language support, voice cloning, and Claude integration via MCP.

What's New in v2.1.6

• Windows: config and settings files with Chinese in them now read correctly, because every text file the package reads or writes is UTF-8.

What's New in v2.1.5

• Emotion without the wait: with --emotion, SenseVoice listens while the main model transcribes. On a Mac Studio, 18 s of audio took 0.72 s instead of 1.08 s.
• Failures say why: every server error logs its traceback, and each request logs how long it took.
• Current recommendations: on NVIDIA and CPU machines, kin audio recommend suggests Qwen3-ASR, Qwen3-TTS, CosyVoice3, Parakeet and SenseVoice.
• Dependencies updated for 143 security advisories. Tests run on Linux, macOS and Windows on every push.

What's New in v2.1.4

• Agents anywhere: kin mcp --http serves the MCP tools to agents on other machines or in the cloud. Audio goes in as a URL or base64, and results come back as download links (setup).
• Streaming speech: /v1/audio/speech sends audio while it's made. On a Mac Studio with Qwen3-TTS, a 23-second reply starts after 0.17 s instead of 4.8 s.
• Access keys: --api-key on kin audio serve and kin mcp --http. An OpenAI SDK's api_key is the key.
• Setup for VS Code, Windsurf and Gemini CLI.

What's New in v2.1.3

• Works with your agents. Set up for each in Use with AI agents:
  • kin mcp gives Claude Code, Claude Desktop, Cursor, Codex and other MCP clients tools to transcribe, speak, clone voices and make music. Setup is one line.
  • kin audio serve --stt … --tts … puts both on one OpenAI-compatible port for agent frameworks.
  • kin skill installs a skill for agents that run commands.
• kin audio tts takes --instruct for tone and --ref-audio to clone a voice.
• Fixes: mp3 uploads to the transcription endpoints, MusicGen on Apple Silicon, and scipy on macOS 27.

What's New in v2.1.2

• Voice cloning over HTTP: /synthesize takes reference_audio (a few seconds of someone talking, as base64 WAV or a path on the server) and reference_text, and a voice-cloning model (kin audio serve qwen3-tts:1.7b-base) says the text in that voice.
• Three new models:
  • higgs-audio:v3 (Higgs TTS 3): made for voice chat, with inline emotion and sound tags and 100+ languages.
  • audio8-tts:0.6b: Mandarin and Cantonese, 44.1 kHz.
  • audio8-asr:infinite: streaming Chinese/English recognition.
• Benchmarked against the defaults, which stay qwen3-asr:1.7b and qwen3-tts:0.6b.

What's New in v2.1.1

• Songs with lyrics: kin audio music generate --model minimax-music3 --lyrics @song.txt — MiniMax Music 3 on MLX (29 s of song in about 3 minutes on an M3 Ultra).
• Music from ComfyUI: ACE-Step 1.5, YuE2, Stable Audio 3 — or any audio blueprint ComfyUI has — run through a local or remote ComfyUI with the weights already installed there.
• kin audio music models shows the same table as kin audio models, with whether each model's weights are installed.

What's New in v2.1.0

• 50 current models, one name each, on any machine. Qwen3-ASR, Fun-ASR-Nano, FireRedASR2, Parakeet, Nemotron, VibeVoice-ASR; Qwen3-TTS, VoxCPM2, CosyVoice3, IndexTTS-2, Fish Audio S2 Pro, Chatterbox and more — picked by Hugging Face downloads, likes and trending. On a Mac a model runs on MLX ([mlx] extra); on CUDA or CPU it runs in its own environment, built automatically the first time.
• Benchmarked on real speech (FLEURS, Mac Studio M3 Ultra). Qwen3-ASR 1.7B beats SenseVoice-Small in Mandarin (9.4 vs 10.5% CER), Cantonese (6.6 vs 8.7%) and English (3.9 vs 8.2% WER) while running faster; Fun-ASR-Nano has the lowest Mandarin CER (8.6%). Qwen3-TTS 0.6B synthesizes 5× faster than real time and handles Chinese–English code-switching that Kokoro misreads. Full tables in the changelog.
• Every model in kin audio models loads, and the HTTP server serves all of them. /synthesize gains instruct for tone and voice design.
• Fixes found on a clean machine: SenseVoice no longer runs pip install on load, Kokoro can't take the server down when its spaCy setup fails, CosyVoice works for the first time, MusicGen plays at the right pitch.

Recommended:

kin audio serve qwen3-asr:1.7b --port 8000 --emotion sensevoice:small   # STT, with SenseVoice's emotion labels
kin audio serve qwen3-tts:0.6b --port 8001                              # TTS, accepts Kokoro voice ids

See CHANGELOG.md for full history.

Features

• STT: Qwen3-ASR, Fun-ASR-Nano, FireRedASR2, GLM-ASR, Audio8 ASR, Parakeet, Canary, Nemotron, Voxtral Realtime, VibeVoice-ASR, Whisper (openai / faster-whisper / whisper.cpp / MLX), SenseVoice, Paraformer, Moonshine — full list
• TTS: Qwen3-TTS, VoxCPM2, CosyVoice3, IndexTTS-2, Fish Audio S2 Pro, Audio8 TTS, Chatterbox, Spark-TTS, OmniVoice, CSM, Dia, Higgs Audio, Kokoro, ChatTTS, F5-TTS and more
• Any platform: one model name everywhere — MLX on Apple Silicon, torch on CUDA/CPU in auto-built isolated environments
• Music Generation: HeartMuLa (multilingual, tag-based), MusicGen
• Voice Activity Detection: Engine-agnostic /vad endpoint via TEN-VAD; or built-in Silero VAD inside faster-whisper
• Chinese Language Support: Optimized models for Mandarin, Cantonese, and mixed Chinese-English
• Voice Cloning: Zero-shot voice cloning with F5-TTS and CosyVoice
• Agents: an MCP server (kin mcp) for Claude Code, Claude Desktop, Cursor, Codex, VS Code and other MCP clients, on this machine or over HTTP for remote ones; an OpenAI-compatible API on one port, with streaming speech and an optional API key; a skill (kin skill) for agents that run commands — setup
• WebSocket Streaming: Real-time transcription and synthesis
• REST API: FastAPI-based server with OpenAPI docs; OpenAI-compatible /v1/audio/transcriptions and /v1/audio/speech
• Subtitle Output: Direct SRT / WebVTT generation from /transcribe?response_format=srt|vtt

Quick Start

# Install (uv recommended)
uv pip install localkin-service-audio

# Get model recommendations for your hardware
kin audio recommend

# View configuration
kin audio config

# Transcribe audio
kin audio transcribe audio.wav

# Text-to-speech
kin audio tts "Hello world"

# Generate music (with Chinese support!)
kin audio music generate "在月光下弹钢琴"  # Chinese lyrics
kin audio music generate "happy wedding" --tags "piano,romantic,wedding" --model heartmula:3b

# Real-time listening (microphone)
kin audio listen

# Voice AI conversation
kin audio listen --llm ollama --tts --stream

# List available models
kin audio models

# Start API server (speech to text and text to speech on one port)
kin audio serve --stt qwen3-asr:1.7b --tts qwen3-tts:0.6b --port 8000

# Start web interface
kin web

Installation

Using uv (recommended — 10x faster)

This project has heavy ML dependencies (~4GB: PyTorch, Whisper, transformers). uv resolves and installs them 10-100x faster than pip.

# Install uv (one-time)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install
uv pip install localkin-service-audio

# Or from source
git clone https://github.com/LocalKinAI/localkin-service-audio.git
cd localkin-service-audio
uv sync

Using in a new terminal: The virtual environment needs to be activated each session:

# Option 1: Activate the venv
source .venv/bin/activate
kin audio models

# Option 2: Use uv run (no activation needed)
uv run kin audio models

To auto-activate, add to your ~/.zshrc or ~/.bashrc:

# Activate .venv automatically when entering a project directory
cd() { builtin cd "$@" && [ -f .venv/bin/activate ] && source .venv/bin/activate; }

Using pip

pip install localkin-service-audio

pip works but is significantly slower due to dependency resolution with large ML packages. Expect 10-30 minutes on first install.

Upgrading

# Upgrade to latest version
uv pip install --upgrade localkin-service-audio

# If upgrading from v2.0.3 or earlier, also upgrade torch (required for v2.0.4+)
uv pip install --upgrade torch torchaudio torchvision

Optional Dependencies

# Chinese language models
uv pip install localkin-service-audio[chinese]

# Voice cloning models
uv pip install localkin-service-audio[cloning]

# MCP server for agents (Claude Code, Cursor, Codex, ...)
uv pip install "localkin-service-audio[mcp]"

# Apple Silicon: run catalog models on MLX (fastest on a Mac). Optional —
# without it they run in auto-built torch environments like everywhere else.
uv pip install "localkin-service-audio[mlx]"

# All features
uv pip install localkin-service-audio[all-new]

Replace uv pip with pip if not using uv.

CLI Usage

Speech-to-Text

# Basic transcription (auto-selects best model)
kin audio transcribe audio.wav

# Specify model
kin audio transcribe audio.wav --model whisper-cpp:base
kin audio transcribe audio.wav --model faster-whisper:large-v3
kin audio transcribe audio.wav --model sensevoice:small  # Chinese

# With language hint
kin audio transcribe audio.wav --language zh

# Output formats
kin audio transcribe audio.wav --format json
kin audio transcribe audio.wav --format srt --timestamps

Text-to-Speech

# Basic synthesis (uses Kokoro with af_heart voice)
kin audio tts "Hello world"

# List all available voices
kin audio tts --model kokoro --list-voices

# American English voices
kin audio tts "Hello world" --voice af_bella       # Bella (Female)
kin audio tts "Hello world" --voice am_adam         # Adam (Male)
kin audio tts "Hello world" --voice af_nova         # Nova (Female)

# British English voices
kin audio tts "Good morning" --voice bf_emma        # Emma (British Female)
kin audio tts "Good morning" --voice bm_george      # George (British Male)

# Chinese (Mandarin) voices
kin audio tts "你好世界" --voice zf_xiaoxiao         # Xiaoxiao (Chinese Female)
kin audio tts "今天天气真好" --voice zm_yunyang       # Yunyang (Chinese Male)

# Japanese voices
kin audio tts "こんにちは" --voice jf_alpha           # Alpha (Japanese Female)
kin audio tts "ありがとう" --voice jm_kumo            # Kumo (Japanese Male)

# French, Spanish, Italian, Hindi, Portuguese
kin audio tts "Bonjour le monde" --voice ff_siwis   # French
kin audio tts "Hola mundo" --voice ef_dora           # Spanish
kin audio tts "Ciao mondo" --voice if_sara           # Italian
kin audio tts "नमस्ते" --voice hf_alpha              # Hindi
kin audio tts "Olá mundo" --voice pf_dora            # Portuguese

# Adjust speech speed (0.5 = slow, 2.0 = fast)
kin audio tts "Hello world" --speed 0.8
kin audio tts "Hello world" --speed 1.5

# Save to file
kin audio tts "Hello world" --output speech.wav

# Save without auto-playing
kin audio tts "Hello world" --output speech.wav --no-play

# CosyVoice for Chinese (voice cloning capable)
kin audio tts "你好世界" --model cosyvoice:300m --voice 中文女

# Qwen3-TTS: Chinese and English mixed in one sentence, tone in words
kin audio tts "今天真不错！" --model qwen3-tts:0.6b --instruct "用开心的语气" -o happy.wav --no-play

# Clone a voice from a 3-10 s recording
kin audio tts "用我的声音说" --model qwen3-tts:1.7b-base --ref-audio me.wav --ref-text "录音里说的话" -o mine.wav --no-play

# Design a voice from a description
kin audio tts "欢迎收听" --model qwen3-tts:1.7b-voicedesign --instruct "沉稳的中年男声，播音腔" -o host.wav --no-play

Music Generation

# MusicGen — text-to-music (small/medium/large)
kin audio music generate "calm piano melody"
kin audio music generate "upbeat electronic" --duration 20 --model musicgen:medium
kin audio music generate "ambient soundscape" -o ambient.wav --device mps

# HeartMuLa — multilingual with Chinese lyrics support
kin audio music generate "在月光下弹钢琴" --model heartmula:3b
kin audio music generate "happy wedding day" --tags "piano,romantic,wedding" --model heartmula:3b --duration 30
kin audio music generate "春天来了，鸟儿在唱歌" --tags "acoustic,happy,upbeat" -o spring.wav

# MiniMax Music 3 — full songs with lyrics (MLX on Apple Silicon, else ComfyUI)
kin audio music generate "中文流行抒情，女声，钢琴" --model minimax-music3 --lyrics @song.txt --duration 60 -o song.flac

# Anything ComfyUI has as an audio blueprint: ACE-Step 1.5, YuE2, Stable Audio 3
kin audio music generate "cinematic rain ambience" --model stable-audio3 --comfyui-url http://box:8188
kin audio music generate "..." --model "comfyui:Text to Music (YuE2)" --param cfg_scale=2.0

# List music models and requirements (includes ComfyUI's, with whether the weights are installed)
kin audio music models
kin audio music models --verbose

ComfyUI models run the blueprint ComfyUI ships for them, on whatever machine runs ComfyUI (--comfyui-url or LOCALKIN_COMFYUI_URL, default http://localhost:8188), using the weights already installed there. --lyrics takes text or @file; --param name=value sets any blueprint input. On Apple Silicon, minimax-music3 defaults to mlx-audio (--backend mlx): through ComfyUI on a Mac its text encoder runs on the CPU, at several seconds per step.

HeartMuLa style tags: piano, acoustic, electric, synthesizer, happy, sad, romantic, calm, upbeat, wedding, ambient, orchestral, rock, pop, jazz, folk, classical, cinematic

Model	Sizes	VRAM	Languages	Duration
MusicGen	small (2GB), medium (4GB), large (16GB)	2–16 GB	English	5–30s
HeartMuLa	3B (6GB), 7B (16GB)	6–16 GB	en, zh, ja, ko, es	5–240s

HeartMuLa setup — auto-installs on first use, or pull in advance:

kin audio pull heartmula:3b

Real-time Listening

# Basic real-time transcription
kin audio listen

# With TTS echo
kin audio listen --tts --tts-model kokoro

# Voice AI with LLM (requires Ollama)
kin audio listen --llm ollama --tts --stream

# Custom models
kin audio listen --model sensevoice:small --language zh --tts --tts-model cosyvoice:300m

# Adjust silence detection
kin audio listen --silence-threshold 0.02 --silence-duration 2.0

Model Management

# List all models with availability status
kin audio models

# Filter by type, language, engine, or tag
kin audio models --type stt
kin audio models --type tts
kin audio models --language zh
kin audio models --engine kokoro
kin audio models --tag voice-cloning
kin audio models --search whisper

# Pull a model
kin audio pull whisper-cpp:base
kin audio pull heartmula:3b

# Remove a model
kin audio rm whisper-cpp:base

# Add a model from a template
kin audio add-model --template whisper_stt --name my-whisper

# Add a model from HuggingFace
kin audio add-model --repo openai/whisper-medium --name whisper-med --type stt

# List available model templates
kin audio list-templates

Model Recommendations

# Get hardware-aware model recommendations
kin audio recommend

# With detailed hardware info
kin audio recommend --verbose

The recommend command detects your hardware (GPU, RAM, CPU) and suggests optimal STT/TTS models for your system.

Configuration

# View configuration overview
kin audio config

# Show configuration file paths
kin audio config --path

# Show all registered models
kin audio config --models

# Initialize config directory with sample config
kin audio config --init

# Change settings
kin audio config set default_tts_model kokoro
kin audio config set default_stt_model faster-whisper:large-v3
kin audio config set api_port 9000
kin audio config set default_device cuda

Configuration files are stored in $LOCALKIN_HOME/ (default: ~/.localkin-service-audio/).

Set LOCALKIN_HOME to relocate all data (cache, config, models) to another disk:

export LOCALKIN_HOME="/path/to/large/disk/.localkin-service-audio"

System Status & Diagnostics

# Check system status (libraries, registry, cache)
kin audio status

# Show cache info
kin audio cache info

# Clear cache for a specific model
kin audio cache clear whisper-large

# Clear all cached models
kin audio cache clear

# Show running LocalKin Audio servers
kin audio ps

API Server

# Start REST API server: one model, or --stt and --tts together
kin audio serve qwen3-tts:0.6b --port 8001
kin audio serve --stt qwen3-asr:1.7b --tts qwen3-tts:0.6b --port 8000

# Start web interface
kin web --port 5000

Supported Models

kin audio models shows all 79 models with real-time availability status:

• ✅ Ready — engine installed, usable now
• 📦 Not installed — strategy code exists, just needs pip install

Current models (catalog)

Use the same name on any machine. Mac (MLX) needs Apple Silicon and the [mlx] extra; torch runs on CUDA, CPU or MPS in an environment built on first use (needs uv; set LOCALKIN_AUTO_INSTALL=0 to refuse, and kin audio pull <model> to build ahead of time). Weights download on first use. Repos and parameters live in catalog.py.

Speech-to-Text

Model	Languages	Notes	Mac (MLX)	CUDA / CPU (torch)
qwen3-asr:0.6b	zh, en, yue, ja, ko, de …	beats Whisper large-v3 on Chinese, 22 dialects, 30 languages	✅	✅
qwen3-asr:1.7b	zh, en, yue, ja, ko, de …	AISHELL-2 WER 2.71 vs 5.06 for Whisper large-v3	✅	✅
fireredasr2:aed	zh, en	lowest Mandarin CER published (AISHELL-1 0.57), 20+ dialects	✅	✅
fun-asr:nano	zh, en, ja	Tongyi's ASR, 7 Chinese dialects and 26 accents	✅	✅
glm-asr:nano	zh, en, yue	Zhipu's ASR, strong on Cantonese and quiet speech	✅	✅
audio8-asr:infinite	zh, en	streaming zh/en, AISHELL-1 CER 1.75 at 480 ms delay	—	✅
confucius4:r2t2	zh, en	NetEase Youdao's Qwen3-ASR-based recognizer	✅	—
moss-transcribe-diarize	zh, en	speaker labels + timestamps	✅	—
whisper-mlx:large-v3-turbo	en, zh, de, es, fr, ja …	Whisper large-v3-turbo on MLX (elsewhere use faster-whisper:large-v3-turbo)	✅	—
nemotron-asr:0.6b	en, zh, es, de, fr, it …	NVIDIA cache-aware streaming ASR, 40 language-locales	✅	✅
voxtral-realtime:4b	en, zh, fr, es, de, it …	Mistral's streaming ASR	✅	✅
vibevoice-asr:9b	en, zh, ja, ko, de, fr …	Microsoft, up to 60 min audio with speakers and hotwords	✅	✅
parakeet:0.6b	en, fr, de, es, pt, it …	NVIDIA, 25 European languages, far faster than Whisper	✅	✅
parakeet:0.6b-v2	en	NVIDIA English ASR (torch: community HF port)	✅	✅
parakeet:1.1b	en	NVIDIA English ASR (torch: community HF port)	✅	✅
canary:1b-v2	en, fr, de, es, pt, it …	NVIDIA transcription + translation, 25 languages (no auto-detect)	✅	✅
canary-qwen:2.5b	en	NVIDIA English ASR leaderboard leader	✅	—

Text-to-Speech

Model	Languages	Notes	Mac (MLX)	CUDA / CPU (torch)
qwen3-tts:0.6b	zh, en, ja, ko, de, fr …	9 preset voices incl. dialects, emotion via instruct	✅	✅
qwen3-tts:1.7b	zh, en, ja, ko, de, fr …	most-downloaded TTS of 2026, emotion via instruct	✅	✅
qwen3-tts:1.7b-voicedesign	zh, en, ja, ko, de, fr …	describe the voice in words (pass instruct)	✅	✅
qwen3-tts:0.6b-base	zh, en, ja, ko, de, fr …	3-second zero-shot voice cloning	✅	✅
qwen3-tts:1.7b-base	zh, en, ja, ko, de, fr …	higher-fidelity zero-shot voice cloning	✅	✅
voxcpm2	zh, en, ja, ko, de, fr …	Seed-TTS zh CER 0.97, 48 kHz, voice design and 9 Chinese dialects	✅	✅
cosyvoice3:0.5b	zh, en, ja, ko, yue, de …	Tongyi, zh CER 1.12, 18+ dialects, instruct control	—	✅
cosyvoice2:0.5b	zh, en, ja, ko, yue, de …	9 languages + 18 Chinese dialects, voice cloning	—	✅
cosyvoice:300m	zh, en, ja, ko, yue	original CosyVoice with 7 preset voices	—	✅
indextts:2	zh, en	bilibili's zero-shot TTS with emotion control	—	✅
indextts:2.5	zh, en	newer IndexTTS, faster and more stable	—	✅
fish-speech:s2-pro	zh, en, ja, ko, de, fr …	inline emotion tags, voice cloning	✅	—
audio8-tts:0.6b	zh, yue, en, ja, ko, de …	Mandarin and Cantonese, zero-shot cloning, 44.1 kHz	✅	—
spark-tts:0.5b	zh, en	gender and pitch control, voice cloning	✅	—
longcat-audiodit:1b	zh, en	Meituan diffusion TTS, zh CER 1.18	✅	—
breeze-tts:2	zh, en	Taiwanese Mandarin + English, voice design	✅	—
confucius4-tts	zh, en, ja, ko, de, fr …	NetEase Youdao, 14 languages	✅	—
moss-tts:nano	zh, en, ja, ko, de, fr …	tiny multilingual cloning TTS	✅	—
moss-tts:local-v1.5	zh, en, ja, ko, de, fr …	31 languages	✅	—
ming-omni-tts:0.5b	zh, en	Ant Group, style control	✅	—
chatterbox:multilingual	en, zh, ja, ko, de, fr …	Resemble AI, 23 languages, emotion exaggeration control	✅	✅
chatterbox:turbo	en	Resemble AI's fast English TTS with paralinguistic tags	✅	✅
omnivoice	zh, en, ja, ko, de, fr …	k2-fsa zero-shot TTS for 600+ languages, nonverbal tags	✅	✅
higgs-audio:v3	zh, en, ja, ko, de, fr …	Boson AI voice-chat TTS, inline emotion/pause/sound tags, 100+ languages	✅	—
higgs-audio:v2	en, zh, ko, de, es	Boson AI expressive TTS, multi-speaker dialogue	✅	✅
outetts:1b	en, zh, ja, ko, de, fr …	Llama-based multilingual TTS	✅	—
voxtral-tts:4b	en, fr, es, de, it, pt …	Mistral, 20 preset voices, no Chinese	✅	—
vibevoice:0.5b	en	Microsoft streaming TTS	✅	—
vibevoice:1.5b	en, zh	Microsoft long-form multi-speaker TTS	—	✅
csm:1b	en	conversational speech model with voice cloning	✅	✅
dia:1.6b	en	multi-speaker dialogue in one pass ([S1]/[S2]), laughter and coughs	✅	✅
orpheus:3b	en	Llama-based emotional TTS, tags	✅	—
kitten-tts:mini	en	under 100 MB	✅	—
soprano:80m	en	tiny, very fast English TTS	✅	—
pocket-tts	en, fr, de, pt, it, es	100M, real-time on CPU, voice cloning	✅	—
irodori-tts	ja	Japanese TTS	✅	—

The torch backends follow each project's documented API. Qwen3-ASR, Qwen3-TTS, Audio8 ASR (MPS) and CosyVoice3 (CPU) have run end to end on the torch path; CUDA runs are awaiting reports — issues welcome. Benchmark results cover 8 STT and 9 TTS models.

Built-in engines

Speech-to-Text

Tip — Voice Activity Detection has two paths since v2.0.12:

1. Inline with transcription — use a faster-whisper:* model and pass enable_vad=true to /transcribe. Silero VAD is bundled inside the engine; transitions are merged into the resulting transcript.
2. Standalone, engine-agnostic — call POST /vad (always available, no model required). Backed by TEN-VAD — 731 KB native macOS arm64 binary, faster transitions than Silero. Returns raw speech segments so you can chunk audio before transcription or use it for diarization-lite workflows. See the /vad endpoint docs.

Model	Engine	Languages	Features	Status
whisper:tiny/base/small/medium/large-v3	OpenAI Whisper	Multilingual	Standard reference	Ready
whisper:large-v3-turbo	OpenAI Whisper	Multilingual	6x faster than large-v3, 809M params	Ready
faster-whisper:tiny/base/large-v3/turbo/distil-large-v3	CTranslate2	Multilingual	4x faster, GPU, native VAD	Ready
faster-whisper:large-v3-turbo	CTranslate2	Multilingual	CTranslate2 turbo variant	Ready
whisper-cpp:tiny/base/small/medium	whisper.cpp	Multilingual	Fast CPU inference	Ready
moonshine:tiny/base	Moonshine	English	5x real-time, ~20MB	Install needed
sensevoice:small	FunASR (Alibaba)	zh, en, ja, ko	15x faster, emotion detection	Install needed
paraformer:zh	FunASR (Alibaba)	Chinese	Fast Chinese ASR	Install needed

Text-to-Speech

Model	Engine	Languages	Features	Status
native	pyttsx3	System	No download needed	Ready
kokoro / kokoro:82m	Kokoro	en, es, fr, hi, it, ja, pt, zh	54 voices, multilingual	Ready
chattts	ChatTTS	zh, en	Conversational, emotion	Install needed
f5-tts	F5-TTS	en, zh	Zero-shot voice cloning	Install needed

Music Models

Model	Engine	Languages	Features	Status
musicgen:small/medium/large	MusicGen (Meta)	English	Text-to-music, 5–30s	Install needed
heartmula:3b/7b	HeartMuLa	en, zh, ja, ko, es	Chinese lyrics, tag control, up to 240s	Install needed

Kokoro Voice Reference

Kokoro supports 54 voices across 9 languages. Voice IDs follow the pattern {lang}{gender}_{name}:

Prefix	Language	Example Voices
af_	American English (Female)	af_heart, af_bella, af_nova, af_sarah, af_sky
am_	American English (Male)	am_adam, am_michael, am_echo, am_puck
bf_	British English (Female)	bf_emma, bf_alice, bf_lily, bf_isabella
bm_	British English (Male)	bm_george, bm_lewis, bm_daniel, bm_fable
zf_	Chinese Mandarin (Female)	zf_xiaoxiao, zf_xiaobei, zf_xiaoni, zf_xiaoyi
zm_	Chinese Mandarin (Male)	zm_yunyang, zm_yunxi, zm_yunjian, zm_yunxia
jf_	Japanese (Female)	jf_alpha, jf_nezumi, jf_gongitsune, jf_tebukuro
jm_	Japanese (Male)	jm_kumo
ff_	French (Female)	ff_siwis
ef_	Spanish (Female)	ef_dora
em_	Spanish (Male)	em_alex
hf_	Hindi (Female)	hf_alpha, hf_beta
hm_	Hindi (Male)	hm_omega, hm_psi
if_	Italian (Female)	if_sara
im_	Italian (Male)	im_nicola
pf_	Portuguese (Female)	pf_dora
pm_	Portuguese (Male)	pm_alex

Python API

from localkin_service_audio import AudioEngine, transcribe, synthesize

# Quick functions
result = transcribe("audio.wav", model="whisper-cpp:base")
print(result.text)

audio = synthesize("Hello world", model="kokoro")
audio.save("output.wav")

# Full engine control
engine = AudioEngine()

# Load and use STT
engine.load_stt("whisper-cpp:base")
result = engine.transcribe("audio.wav", language="en")
print(f"Text: {result.text}")
print(f"Language: {result.language}")

# Load and use TTS - English
engine.load_tts("kokoro")
audio = engine.synthesize("Hello world", voice="af_heart")
audio.save("english.wav")

# TTS - Chinese (auto-selects Chinese pipeline)
audio = engine.synthesize("你好世界", voice="zf_xiaoxiao")
audio.save("chinese.wav")

# TTS - Japanese
audio = engine.synthesize("こんにちは世界", voice="jf_alpha")
audio.save("japanese.wav")

# TTS - with speed control
audio = engine.synthesize("Hello", voice="am_adam", speed=0.8)
audio.save("slow.wav")

# List available voices
voices = engine.list_voices()
for v in voices:
    print(f"{v.id}: {v.name} ({v.language}, {v.gender})")

# Voice cloning (with supported models)
engine.load_tts("f5-tts")
audio = engine.clone_voice(
    reference_audio="reference.wav",
    text="Text to speak in cloned voice"
)

Use with AI agents

MCP (Claude Code, Claude Desktop, Cursor, Codex, …)

Install with the mcp extra (on Apple Silicon add mlx, so the models run on the GPU):

pip install "localkin-service-audio[mcp]"        # Apple Silicon: "localkin-service-audio[mcp,mlx]"

Then register kin mcp with your client; it starts the server itself.

Claude Code

claude mcp add localkin-audio -- kin mcp

Claude Desktop (~/Library/Application Support/Claude/claude_desktop_config.json on macOS, %APPDATA%\Claude\claude_desktop_config.json on Windows) and Cursor (~/.cursor/mcp.json):

{
  "mcpServers": {
    "localkin-audio": { "command": "kin", "args": ["mcp"] }
  }
}

Codex (~/.codex/config.toml):

[mcp_servers.localkin-audio]
command = "kin"
args = ["mcp"]

Windsurf (~/.codeium/windsurf/mcp_config.json) and Gemini CLI (~/.gemini/settings.json) take the same mcpServers block as Claude Desktop above. VS Code (.vscode/mcp.json, or MCP: Open User Configuration) names it servers:

{
  "servers": {
    "localkin-audio": { "type": "stdio", "command": "kin", "args": ["mcp"] }
  }
}

To skip installing, let uv fetch it: use the command uvx with the arguments ["--from", "localkin-service-audio[mcp]", "kin", "mcp"] ([mcp,mlx] on Apple Silicon).

Tool	What it does
transcribe_audio	Audio file → text, with timestamps; detects the language
synthesize_speech	Text → audio file. voice, instruct for tone ("用开心的语气"), language, speed
clone_voice	Text in the voice of a 3–10 s recording (reference_text makes it closer)
generate_music	Description (and lyrics, for song models) → music file
list_models, list_voices, get_audio_info	What can run, which voices a model has, a file's duration and format

Default models:

Kind	Default model
Transcription	qwen3-asr:1.7b, or qwen3-asr:0.6b under 16 GB of RAM
Speech	qwen3-tts:0.6b
Cloning	qwen3-tts:1.7b-base, or qwen3-tts:0.6b-base under 16 GB of RAM
Music	musicgen:small

• Where the models run: with the mlx extra on Apple Silicon they run on MLX. Elsewhere they run in an environment built on first use, which needs uv.
• Choosing other models: pass --stt, --tts, --clone or --music (e.g. "args": ["mcp", "--tts", "kokoro"]), or set LOCALKIN_STT_MODEL, LOCALKIN_TTS_MODEL, LOCALKIN_CLONE_MODEL or LOCALKIN_MUSIC_MODEL. On a CPU-only machine, --stt faster-whisper:large-v3-turbo --tts kokoro is much faster.
• First use: a model downloads the first time it's used. Run kin audio pull qwen3-tts:0.6b (and the others) beforehand if your client times out on that first call.
• Output files: these go to ~/.localkin-service-audio/outputs/ unless the agent passes output_path.

Remote MCP (agents on other machines or in the cloud)

Run the MCP server over HTTP on the machine with the models:

kin mcp --http --host 0.0.0.0 --api-key "$(openssl rand -hex 16)"     # MCP at http://<host>:8765/mcp

Then give your client the URL and the key:

claude mcp add --transport http localkin-audio http://192.168.0.21:8765/mcp \
  --header "Authorization: Bearer YOUR_KEY"

Other clients that support remote MCP take the same URL plus an Authorization: Bearer YOUR_KEY header. For example, Cursor uses {"url": "...", "headers": {...}} and VS Code uses "type": "http".

How remote mode differs:

• Audio in: the server can't see the client's files, so audio comes in as audio_url / reference_url (http or https) or audio_base64 / reference_base64. Paths only reach files the server produced itself.
• Results out: each result comes back as a url to download. Pass return_base64: true to get the audio in the reply as well. output_path isn't accepted.
• Access: downloads and /mcp need the key; /health doesn't.
• Advertised address: the server announces its LAN address. Behind a reverse proxy or tunnel, set the address clients actually use with --public-url https://audio.example.com.

OpenAI-compatible HTTP API (LangChain, LiveKit Agents, Pipecat, Open WebUI, …)

Serve a speech-to-text and a text-to-speech model on one port:

kin audio serve --stt qwen3-asr:1.7b --tts qwen3-tts:0.6b --port 8000

and point any OpenAI client at it:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="local")   # any key, unless the server has --api-key
with open("question.mp3", "rb") as f:
    text = client.audio.transcriptions.create(model="whisper-1", file=f, language="zh").text
client.audio.speech.create(model="tts-1", voice="vivian", input="你好！", instructions="用开心的语气",
                           response_format="mp3").write_to_file("answer.mp3")

/v1/audio/transcriptions

• language.
• response_format: json, text, srt, vtt or verbose_json.
• Uploads can be wav, flac, mp3 or ogg; m4a and webm work once ffmpeg is installed.

/v1/audio/speech

• voice: a voice from GET /voices. OpenAI's own voice names fall back to the model's default voice.
• instructions: the tone to use.
• speed.
• response_format: wav (the default here), mp3, opus, flac or pcm.

Streaming speech

stream_format: "audio" sends the speech while it's being made, and "sse" sends OpenAI's speech.audio.delta events. Either works with response_format pcm, wav or mp3; pcm streams by default, as OpenAI's does. Models that can stream within a sentence (Qwen3-TTS) send audio every 0.6 s of speech; the rest go one sentence at a time. Measured on a Mac Studio, a 23-second reply began after 0.17 s instead of 4.8 s.

with client.audio.speech.with_streaming_response.create(
        model="tts-1", voice="vivian", input=long_text, response_format="pcm") as response:
    for chunk in response.iter_bytes():
        player.write(chunk)          # 16-bit mono, 24 kHz

Both endpoints

• /v1/models lists the two models. The model field in a request is ignored; the server's own models always answer.
• The server listens on 127.0.0.1. To reach it from other machines, add --host 0.0.0.0 together with --api-key KEY (or set LOCALKIN_API_KEY). Requests then need Authorization: Bearer KEY, which is what an OpenAI SDK sends as its api_key. /health and the docs stay open.

Agent skill (Claude Code, OpenClaw, …)

For agents that can run commands, a skill teaches them to use kin:

• which model fits which job
• the commands for transcribing, speaking, cloning, designing a voice, music and running a server

kin skill                            # into ~/.claude/skills, for Claude Code in every project
kin skill --dir .claude/skills       # only this project
kin skill --dir ~/.openclaw/skills   # another agent's skills folder
kin skill --print                    # the text itself, to paste into AGENTS.md or a prompt

The same skill is in openclaw-skill/ with OpenClaw's metadata.

REST API

Start the server:

kin audio serve --stt qwen3-asr:1.7b --tts qwen3-tts:0.6b --port 8000

Endpoints

POST /transcribe - Transcribe audio

# Basic (language is a query parameter; omit it to auto-detect)
curl -X POST "http://localhost:8000/transcribe?language=en" \
  -F "file=@audio.wav"

With SenseVoice as the model, or any model served with --emotion sensevoice:small, the JSON also carries emotion (happy / sad / angry / neutral / …) and audio_events (laughter, applause, …). Treat the label as a hint.

Optional query parameters (added in v2.0.11):

Param	Type	Default	Notes
language	string	auto	BCP-47 language code, e.g. en, zh
enable_vad	bool	true	Skip silence via VAD (faster-whisper only)
timestamps	bool	false	Include segment timings in JSON response
response_format	enum	json	json | text | markdown | srt | vtt
chunk_length_s	int	engine default	VRAM tuning for long audio

# Markdown transcript with timestamps
curl -X POST 'http://localhost:8000/transcribe?response_format=markdown' \
  -F 'file=@meeting.wav'

# SRT subtitles for video captioning
curl -X POST 'http://localhost:8000/transcribe?response_format=srt' \
  -F 'file=@video.wav' > captions.srt

# Low-VRAM long-audio: VAD + smaller chunks
curl -X POST 'http://localhost:8000/transcribe?chunk_length_s=15&enable_vad=true' \
  -F 'file=@long.wav'

# JSON with segment timestamps (no shape change to existing callers
# unless you opt in with timestamps=true)
curl -X POST 'http://localhost:8000/transcribe?timestamps=true' \
  -F 'file=@audio.wav'

POST /synthesize - Synthesize speech (returns audio/wav)

curl -X POST "http://localhost:8001/synthesize" \
  -H "Content-Type: application/json" \
  -d '{"text": "今天天气很好", "speaker": "vivian", "instruct": "用开心的语气"}' \
  --output speech.wav

Field	Default	Notes
text	—	Required
speaker	model default	A voice id from GET /voices. Kokoro ids (zf_xiaoxiao, af_heart…) are mapped to the nearest voice on other models
language	from the text	ISO code, e.g. zh, en
speed	1.0	0.5–2.0
instruct	—	Tone on Qwen3-TTS CustomVoice ("用开心的语气"), a voice description on VoiceDesign models; ignored elsewhere
reference_audio	—	A voice to clone: a few seconds of it as base64 WAV, or a path on the server's machine. Only voice-cloning models take it (qwen3-tts:1.7b-base, higgs-audio:v3, …); others answer 400
reference_text	—	What reference_audio says; models that read it along clone more closely

GET /voices - Voices of the TTS model

curl "http://localhost:8001/voices"
# {"model": "qwen3-tts:0.6b", "multilingual": true, "default_voice": "vivian",
#  "voices": [{"id": "vivian", "name": "Vivian · 明亮女声", "language": "zh", "gender": "female"}, ...]}

multilingual: true means every voice reads every language the model supports, so send mixed-language text in one request rather than splitting it between voices (which single-language Kokoro voices need).

POST /vad - Detect speech segments (no transcription)

Engine-agnostic Voice Activity Detection via TEN-VAD (731 KB native macOS arm64 binary, ~0.016 RTF on M1, ~100-300 ms faster transitions than Silero). Useful for chunking long audio before transcription, or for VAD-only workflows.

# Install the optional VAD extra first
pip install 'localkin-service-audio[vad]'

curl -X POST "http://localhost:8000/vad" \
  -F "file=@meeting.wav"

# Output:
# {
#   "backend": "ten-vad",
#   "duration": 132.4,
#   "speech_segments": [
#     {"start": 1.2, "end": 5.8, "duration": 4.6},
#     ...
#   ],
#   "total_speech_duration": 48.3
# }

Tunable parameters (all optional query strings):

Param	Default	Effect
backend	ten-vad	VAD backend (currently only one supported)
threshold	0.5	0.0-1.0 speech-probability cutoff
min_speech_duration_ms	200	Drop speech runs shorter than this
min_silence_duration_ms	200	Merge runs separated by less silence
speech_pad_ms	100	Pad each kept segment by this much

GET /models - List models

curl "http://localhost:8000/models"

GET /health - Readiness. 503 with the missing package when the model's backend isn't installed; loaded: true once the model is in memory (serve preloads by default).

WebSocket /stream - Real-time transcription

const ws = new WebSocket("ws://localhost:8000/stream");
ws.send(audioChunk);
ws.onmessage = (e) => console.log(JSON.parse(e.data).text);

Configuration

Environment Variables

# Base directory for all data (cache, config, models)
export LOCALKIN_HOME="/Volumes/Data/.localkin-service-audio"

# Override individual directories
export LOCALKIN_CACHE_DIR="/tmp/my-cache"
export LOCALKIN_CONFIG_DIR="/path/to/config"
export LOCALKIN_MODELS_DIR="/path/to/models"

# Default engine settings
export LOCALKIN_DEFAULT_STT="faster-whisper:large-v3"
export LOCALKIN_DEFAULT_TTS="kokoro"
export LOCALKIN_DEVICE=cuda  # or cpu, mps, auto

# API server
export LOCALKIN_API_HOST="127.0.0.1"
export LOCALKIN_API_PORT="8000"

Custom Models

Create $LOCALKIN_HOME/models.json (default: ~/.localkin-service-audio/models.json):

{
  "models": {
    "my-custom-model": {
      "type": "stt",
      "engine": "whisper",
      "model_size": "base",
      "languages": ["en", "zh"],
      "description": "My custom model"
    }
  }
}

Architecture

LocalKin Audio v2.0 uses a modular architecture:

• Strategy Pattern: Pluggable STT/TTS engines
• Facade Pattern: AudioEngine provides unified interface
• Registry Pattern: Centralized model configuration
• Singleton Pattern: Shared engine instance

localkin_service_audio/
├── core/
│   ├── audio_processing/
│   │   ├── engine.py          # AudioEngine facade
│   │   ├── stt/               # STT strategies
│   │   │   ├── base.py
│   │   │   ├── whisper_strategy.py
│   │   │   ├── sensevoice_strategy.py
│   │   │   └── ...
│   │   └── tts/               # TTS strategies
│   │       ├── base.py
│   │       ├── kokoro_strategy.py
│   │       ├── cosyvoice_strategy.py
│   │       └── ...
│   ├── config/
│   │   └── model_registry.py  # Model registry
│   └── types.py               # Core dataclasses
├── cli/                       # Click CLI
├── api/                       # FastAPI server
├── mcp/                       # MCP server
└── ui/                        # Web interface

Development

# Clone repository
git clone https://github.com/LocalKinAI/localkin-service-audio.git
cd localkin-service-audio

# Install with dev dependencies (uv recommended)
uv venv && source .venv/bin/activate
uv pip install -e ".[dev]"

# Or with pip
pip install -e ".[dev]"

# Run tests
pytest tests/

# Run linting
ruff check .
black --check .

Tip: With uv, you can skip activation and run commands directly:

uv run kin audio models
uv run pytest tests/

Troubleshooting

Model Loading Errors

# Check model is registered
kin audio models

# Pull the model
kin audio pull whisper-cpp:base

# Check system info
kin info --verbose

PyTorch Version

Requires torch >= 2.6.0. Older versions will fail to load models that only ship .bin weights (e.g. MusicGen medium/large) due to a torch.load security check (CVE-2025-32434).

# Check your version
python -c "import torch; print(torch.__version__)"

# Upgrade if needed (keep torchvision in sync)
pip install "torch>=2.6.0" "torchaudio>=2.6.0" "torchvision>=0.21"

macOS 27: __thread_bss / scipy import error

scipy 1.15.3, the last release with Python 3.10 wheels, fails to load on macOS 27. Create the environment with Python 3.11+:

uv venv -p 3.11 && uv pip install -e ".[sensevoice]"

numpy/pandas Binary Incompatibility

If you see numpy.dtype size changed, may indicate binary incompatibility, pandas or scikit-learn was compiled against a different numpy version:

# Fix: force-reinstall the affected packages
uv pip install --force-reinstall numpy pandas scikit-learn

# Or nuke and rebuild the venv
rm -rf .venv && uv venv && uv pip install localkin-service-audio

CUDA/GPU Issues

# Force CPU
kin audio transcribe audio.wav --device cpu

# Check PyTorch CUDA
python -c "import torch; print(torch.cuda.is_available())"

HeartMuLa on Apple Silicon (MPS)

HeartMuLa 3B requires ~12-14GB. On a 16GB Mac, close memory-heavy apps before running. The codec runs on CPU automatically (shared unified memory, no performance impact).

# If you hit OOM, try shorter duration
kin audio music generate "prompt" --model heartmula:3b --duration 5

# Or force CPU (slower but more stable memory management)
kin audio music generate "prompt" --model heartmula:3b --device cpu

Chinese Model Dependencies

SenseVoice and Paraformer run on FunASR, which is an optional extra — it pulls in modelscope and a sizeable tree, so a Whisper-only install shouldn't have to download it.

# Install the extra (declared in pyproject.toml)
uv pip install --project . -e ".[sensevoice]"

# Or plain pip
pip install funasr modelscope

# Then use Chinese models
kin audio transcribe audio.wav --model sensevoice:small

# Or serve them
kin audio serve sensevoice:small --port 8000

Without the extra, the server still starts and /health still reports {"status": "healthy"} — the failure only shows up on the first /transcribe, as funasr not installed. The model weights are downloaded separately and being present is not enough on its own.

License

MIT License - see LICENSE file.

Acknowledgments

• OpenAI Whisper
• faster-whisper
• whisper.cpp
• FunASR
• Kokoro TTS
• CosyVoice
• MusicGen
• HeartMuLa