voice-coach-mcp

Give Claude ears. A local MCP server that listens to you sing, speak and read aloud, measures what it hears, and lets Claude coach your voice.

Claude cannot process audio. This server records from your microphone, analyzes the audio on your computer, and returns compact measurements that Claude turns into concrete coaching:

A bundled skill turns Claude into a coach: it runs warm-up → exercise → analysis → feedback sessions, gives one concrete focus per take, and keeps a progress log across sessions. Claude replies in your language.

• Local and private: audio never leaves your computer (unless you opt into the Azure backend).
• Free: open-source models (faster-whisper, wav2vec2) and no API keys.
• Tools measure, Claude interprets: every response is compact JSON (≤ 2 KB) with confidence and warnings, so Claude never coaches on bad audio.

Install

You need uv and, for pronunciation, espeak-ng:

brew install uv espeak-ng ffmpeg        # macOS (ffmpeg is optional: m4a/aac input)

On Linux: install uv, then sudo apt install espeak-ng ffmpeg libportaudio2. Pronunciation analysis on macOS needs Apple Silicon and macOS 14 or later (a PyTorch requirement).

Claude Code (plugin: server + skill)

claude plugin marketplace add FabianIMV/voice-coach-mcp
claude plugin install voice-coach@voice-coach

Or, inside a session: /plugin install voice-coach --marketplace FabianIMV/voice-coach-mcp.

The first start installs the Python dependencies with uv, which can take a minute or two (the models download later, on first use). If /mcp shows voice-coach as failed right after installing, reconnect it from /mcp once the installation finishes.

Claude Desktop (extension)

Download voice-coach.mcpb from the latest release and double-click it (or drag it into Claude Desktop). For the coaching skill, also download voice-coach-skill.zip and upload it in Settings → Capabilities → Skills.

Any MCP client (manual)

claude mcp add voice-coach --scope user -- uvx --from "voice-coach-mcp[speech,pronunciation]" voice-coach-mcp

For Claude Desktop's claude_desktop_config.json, use absolute paths (Desktop does not inherit your shell PATH; which uvx prints it):

{
  "mcpServers": {
    "voice-coach": {
      "command": "/opt/homebrew/bin/uvx",
      "args": ["--from", "voice-coach-mcp[speech,pronunciation]", "voice-coach-mcp"]
    }
  }
}

Optional: run uvx --from "voice-coach-mcp[speech,pronunciation]" voice-coach-mcp download-models once, so the first call does not wait for the models (~480 MB Whisper, ~1.3 GB phoneme model).

Microphone permission (macOS)

macOS grants the microphone to the app that launches the server: Claude Desktop, or the terminal running Claude Code (Terminal, iTerm, Ghostty, VS Code…). If permission is denied, macOS returns silence instead of an error and record_audio reports no_microphone_permission. Fix it in System Settings → Privacy & Security → Microphone and restart that app.

Use it

Just talk to Claude:

• "I want to practice singing in tune. I'm a beginner."
• "Record 30 seconds of me presenting my project and tell me how many filler words I use."
• "Help me pronounce 'th'. Give me sentences to read and score them."
• "Analyze this voice memo against melody.mid" (any wav/mp3/m4a path works).
• "How am I doing?" (Claude reads your progress log).

Real sessions in Claude Code: a 60-second talk about work, and a singing baseline on one held note.

Guided sessions are also available as MCP prompts: singing_session, pronunciation_session and speaking_session (in Claude Code: /mcp__voice-coach__singing_session).

Tools

Audio can be a path or the name returned by record_audio. When a result is too long, the most relevant items are returned and everything else is saved next to the audio (full_details).

{"audio":"examples/audio/c_major_scale.wav","duration_s":7.36,"mode":"equal_temperament",
 "summary":{"notes":8,"range":"C4–C5","pct_within_25_cents":75.0,"mean_deviation_cents":-7.8,
   "mean_abs_deviation_cents":13.6,"tendency":"centered","drift_cents":-1.0,
   "stability":{"stable":7,"vibrato":1,"unstable":0}},
 "notes":[{"t":0.4,"dur":0.62,"note":"C4","cents":0,"stability":"stable"},
   {"t":1.84,"dur":0.62,"note":"E4","cents":-30,"stability":"stable"},
   {"t":4.72,"dur":0.62,"note":"B4","cents":-42,"stability":"stable"},
   {"t":5.44,"dur":1.43,"note":"C5","cents":-2,"stability":"vibrato","vibrato_hz":5.5,"vibrato_pp_cents":84}, "…"],
 "confidence":{"value":1.0,"level":"high"},"warnings":[]}

{"backend":"wav2vec2-local",
 "summary":{"overall_score":90,"words":10,"words_with_errors":3,
   "errors":{"substitution":3,"deletion":0,"insertion":0},
   "frequent_confusions":[{"expected":"θ","detected":"s","count":2},{"expected":"v","detected":"b","count":1}]},
 "words":[{"word":"think","score":75,"t":0.56,"expected":"θ ɪ ŋ k","detected":"s ɪ ŋ k",
     "errors":[{"type":"substitution","expected":"θ","detected":"s"}]},
   {"word":"very","score":75,"t":0.96,"expected":"v ɛ ɹ i","detected":"b ɛ ɹ i",
     "errors":[{"type":"substitution","expected":"v","detected":"b"}]}, "…"],
 "confidence":{"value":1.0,"level":"high"},"warnings":[]}

{"language":"en",
 "pace":{"words":36,"speaking_time_s":14.9,"words_per_minute":145,"wpm_excluding_long_pauses":165},
 "pauses":{"threshold_s":0.7,"count":2,"total_s":1.9,"max_s":1.03,"per_minute":8.1,
   "items":[{"t":4.42,"dur":1.03,"after":"speaking"},{"t":12.1,"dur":0.83,"after":"normal"}]},
 "fillers":{"total":6,"per_minute":24.2,"by_type":{"um":1,"so":1,"uh":1,"you know":1,"like":1,"i mean":1}},
 "pitch":{"median_f0_hz":101,"std_semitones":1.53,"monotone":true,"monotone_stretches":[{"t":0.0,"end":12.0}]},
 "volume":{"mean_dbfs":-21.2,"std_between_words_db":2.4,"low_variation":true},
 "confidence":{"value":1.0,"level":"high"},"warnings":[]}

All example outputs are real responses from the server over stdio, regenerated by examples/generate_examples.py on synthetic audio; see examples/outputs/.

How it works

• Pitch: pYIN (librosa) finds voiced frames; each frame is refined from its harmonic peaks with parabolic interpolation (~1 cent accuracy from C2 to C6). Notes are split at silences, energy dips and sustained pitch changes; vibrato does not split notes. With a MIDI reference, the octave and offset are found by cross-correlation, notes are aligned with Needleman–Wunsch, and a tempo line t_sung = a·t_ref + b is fitted to judge each onset.
• Speech: faster-whisper transcribes with word timestamps (an initial prompt keeps fillers). Pauses are measured from the audio energy, because Whisper stretches word endings into silence. Ambiguous fillers ("like", "so") only count when isolated by a comma or a pause.
• Pronunciation: phonemizer + espeak-ng give the expected IPA phonemes; the facebook/wav2vec2-lv-60-espeak-cv-ft phoneme model gives what was said; both are aligned by edit distance. Normal American English variants (flap, schwa, linking r, glottal stop) are not counted as errors.
• Quality first: clipping, background noise, silence and missing microphone permission are reported as warnings and lower the confidence.

Known limitations: one voice at a time (accompaniment degrades pitch detection); two identical notes sung in perfect legato count as one; the phoneme model uses sentence context and may occasionally "correct" a mispronounced word; pronunciation assessment is English-only.

Privacy and network access

Everything the plugin runs, writes and downloads:

• Runs: uv starts the server from the plugin folder (uv run --frozen, pinned by uv.lock); the server uses espeak-ng for expected phonemes and ffmpeg only to read m4a/aac files.
• Microphone: only while record_audio runs, for the number of seconds Claude asks for.
• Writes: recordings and progress.md in ~/VoiceCoach (or VOICE_COACH_DIR), and an analysis detail file <audio>.<tool>.json next to each analyzed audio (or in ~/VoiceCoach/analysis if that folder is read-only). Analyzing the same audio again replaces its detail file; nothing else is overwritten or deleted.
• Downloads: the Python dependencies from PyPI on first start, then the models from Hugging Face on first use: the faster-whisper model (Systran/faster-whisper-small by default) and facebook/wav2vec2-lv-60-espeak-cv-ft. After that, models load from the local cache.
• Sends: nothing by default. No telemetry, no accounts, no API keys. Only if you set VOICE_COACH_PRONUNCIATION_BACKEND=azure with your own AZURE_SPEECH_KEY and AZURE_SPEECH_REGION does assess_pronunciation send that audio and the expected text to Azure Speech in your region.

Configuration

Development

git clone https://github.com/FabianIMV/voice-coach-mcp.git && cd voice-coach-mcp
uv sync --extra speech --extra pronunciation
uv run pytest                 # fast suite: synthetic audio, simulated microphone
uv run pytest -m models       # real Whisper and wav2vec2 models on synthetic speech
uv run ruff check . && uv run ruff format --check .

See CONTRIBUTING.md for the design principles and the release process.

src/voice_coach_mcp/
  server.py         MCP tools and prompts (thin layer)
  audio/            loading, recording and quality checks
  singing/          pitch tracking, music theory, MIDI reference, pitch accuracy
  speech/           transcription, speech metrics, filler lists
  pronunciation/    expected phonemes, alignment, wav2vec2 and Azure backends
  output.py         2 KB response budget and details file
  progress.py       progress log
skills/voice-coach/ coaching skill (also bundled in the Python package)
examples/           synthetic audio and real tool outputs

License and credits

MIT © FabianIMV. Built on faster-whisper, librosa, wav2vec2 phoneme recognition and the official MCP Python SDK. The optional pronunciation extra uses phonemizer and espeak-ng, both GPL-3.0. Example voices were synthesized with espeak-ng and Piper (public-domain LJSpeech voice).