Audio Intelligence MCP Server

Latest release notes: CHANGELOG.md · ▶ Watch with sound (MP4, 29 s)

A local-first Model Context Protocol server that lets AI agents analyze audio: transcripts with word timings, speakers, sound events, acoustic measurements, quality diagnosis, timelines, evidence-grounded answers, comparisons, library search, streaming, and exports.

Everything runs on your machine. Audio never leaves it. An LLM is optional, off by default, and only ever receives text (transcript snippets and event labels); every response reports data_sent_off_machine.

Every tool returns the same JSON envelope:

{
  "status": "success | partial | processing | error",
  "audio_id": "aud_0123456789abcdef",
  "result": { "...": "tool-specific (analyze_audio uses \"analysis\")" },
  "warnings": [{"code": "OVERLAP_UNSUPPORTED", "message": "..."}],
  "errors": [{"code": "STAGE_FAILED", "stage": "events", "message": "...", "retryable": false}],
  "processing": {"duration_seconds": 2.4, "stages_run": ["vad", "transcription"], "stages_skipped": [],
                 "cache_hits": [], "models_used": [{"stage": "transcription", "name": "faster-whisper",
                 "version": "small", "local": true}], "data_sent_off_machine": false, "schema_version": "1.0"}
}

Validation failures use {"status": "error", "error": {"code", "message", "retryable"}}. Times are seconds (3 decimals), levels dBFS, frequencies Hz. confidence appears only when a model or rule produced it, always with confidence_kind (model_probability or heuristic; none are calibrated).

Requirements

• Windows, macOS or Linux (CI runs the test suite on all three)
• Python 3.12 or 3.13 and uv
• ffmpeg and ffprobe on PATH (or set ffmpeg_path / ffprobe_path):
  • Windows: winget install Gyan.FFmpeg
  • macOS: brew install ffmpeg
  • Debian/Ubuntu: sudo apt-get install ffmpeg
• CPU is enough. CUDA is used automatically when available (device = "auto").

Install

From PyPI with uv pip, into a virtual environment:

uv venv --python 3.12
uv pip install "audio-intelligence-mcp[models]"            # local models (faster-whisper, PANNs, SpeechBrain, silero-vad, MiniLM)
uv pip install "audio-intelligence-mcp[models,llm]"        # + optional Claude backend (anthropic SDK)
uv pip install "audio-intelligence-mcp[models,pyannote]"   # + optional pyannote diarization (needs HF_TOKEN)

Then start the server with .venv/bin/audio-mcp (macOS/Linux) or .venv\Scripts\audio-mcp (Windows), or activate the environment first and run audio-mcp. Plain pip install "audio-intelligence-mcp[models]" works the same way.

Run without installing anything (uv fetches and caches the package):

uvx --from "audio-intelligence-mcp[models]" audio-mcp

From source:

git clone https://github.com/DNSdecoded/audio-intelligence-mcp.git
cd audio-intelligence-mcp
uv sync --extra models            # local models (faster-whisper, PANNs, SpeechBrain, silero-vad, MiniLM)
uv sync --extra models --extra llm        # + optional Claude backend (anthropic SDK)
uv sync --extra models --extra pyannote   # + optional pyannote diarization (needs HF_TOKEN)

Without --extra models the server still starts: model-backed stages are reported in stages_skipped with the reason, and the DSP tools (features, quality, integrity, comparison) work.

First-run downloads (cached in models_dir, then fully offline): Whisper small ≈ 460 MB, PANNs CNN14 ≈ 330 MB, SpeechBrain ECAPA ≈ 80 MB, all-MiniLM-L6-v2 ≈ 90 MB. Silero VAD ships inside its package.

Connect a client

Claude Code (from PyPI, no install step):

claude mcp add audio-intel -- uvx --from "audio-intelligence-mcp[models]" audio-mcp

Claude Code (source checkout):

claude mcp add audio-intel -- uv --directory /path/to/audio-intelligence-mcp run audio-mcp

Claude Desktop (claude_desktop_config.json), from PyPI:

{
  "mcpServers": {
    "audio-intel": {
      "command": "uvx",
      "args": ["--from", "audio-intelligence-mcp[models]", "audio-mcp"],
      "env": {"AUDIO_MCP_ALLOWED_ROOTS": "[\"/path/to/your/audio\"]"}
    }
  }
}

From a source checkout, use "command": "uv" with "args": ["--directory", "/path/to/audio-intelligence-mcp", "run", "audio-mcp"] instead.

On Windows, use forward slashes or escaped backslashes in JSON paths ("C:/audio" or "C:\audio").

Only files under allowed_roots can be read (default: your home directory). Paths are resolved, symlinks included, before the check.

Tools

Full schemas and example success/error responses: docs/tools.md. What is implemented, partial, or future: docs/capabilities.md.

Example workflows

Meeting. analyze_audio {"source": {"type": "file_path", "path": "/audio/standup.m4a"}, "goal": "meeting"} plans transcription, language, diarization, events, summary, timeline and uncertainty. Then reuse the returned audio_id (cached, nothing recomputed): ask_audio {"source": {"type": "audio_id", "audio_id": "aud_..."}, "question": "When does someone mention the deadline?"} and export_audio_analysis {"audio_id": "aud_...", "format": "srt"}.

Machine baseline. compare_audio {"sources": [<today.wav>], "baseline": [<healthy_1.wav>, <healthy_2.wav>]} returns per-feature z-scores and the seconds where the target departs from the baseline distribution; interpretations are labelled as such and never stated as causes.

Environmental recording. detect_audio_events {"source": ..., "categories": ["animal", "vehicle"]}, then get_audio_timeline and ask_audio "What happens immediately before the alarm sounds?".

Library. search_audio_library {"action": "index", "paths": ["/audio/calls"]}, then {"action": "search", "query": "refund request", "filters": {"language": "en"}} or search by an example clip with "query_audio".

Streaming. analyze_audio_stream with start (chunk_format: "pcm" + rate/channels/sample format, or "file"), repeated push (base64 data or path), then finish for the full analysis and a reconciliation of provisional vs final results.

Configuration

TOML file (AUDIO_MCP_CONFIG=/path/config.toml, see config.example.toml) and/or env vars AUDIO_MCP_<NAME>. Full table: docs/configuration.md. Key settings:

Privacy

• Audio is processed locally and never sent anywhere.
• The only outbound traffic is (a) model downloads on first use (Hugging Face, Zenodo, GitHub) and (b) the LLM, if you enable one. A local Ollama/OpenAI-compatible server on localhost counts as local; any other host sets processing.data_sent_off_machine: true on that response.
• Stored audio copies and cached results live under data_dir and are swept after retention_ttl_hours (retention = "none" deletes everything at shutdown). The search library is persistent (library_retention = "keep") and stores derived data and file paths, never audio.
• Exports written by export_audio_analysis to export_dir are yours to keep: retention never deletes them. They are redacted when the call sets redact: true or the server sets redact_outputs = true.
• Logs go to stderr (and log_file) and never contain transcript text unless log_sensitive = true.
• Integrity indicators are signal observations only; the server never states that audio is authentic, edited or synthetic, and never infers emotion.

Supported input

Any audio ffmpeg can decode: WAV, FLAC, MP3, OGG/Opus, M4A/AAC, WebM, AIFF, WMA, and more; any sample rate and channel count (models get a 16 kHz mono mix; measurements use the native signal). Sources: file_path, base64, url (off by default), audio_id. Playlist and concat inputs (HLS .m3u8, ffconcat, DASH, image sequences) are rejected because they make ffmpeg read other files.

Limits and memory: audio longer than max_duration_seconds (default 7200 s = 2 h) is rejected. Decoded audio is held in RAM as float32 at the native rate plus a 16 kHz mono copy: about 1.6 GB per hour of 48 kHz stereo (about 0.9 GB per hour of 44.1 kHz mono), with transient peaks of roughly 2-3x that during decoding and feature extraction. The two most recently used recordings stay decoded in memory. Raise the limit only with RAM to match.

Language coverage follows Whisper (≈ 99 languages); translation is to English only.

Evaluation

Measured on public datasets with the default local models on a 4-core CPU (Whisper small int8, PANNs CNN14, ECAPA). Full tables, per-file JSON and before/after numbers: docs/eval/RESULTS.md; method, datasets and caveats: docs/evaluation.md.

Low-resource languages (Arabic, Hindi, Telugu) are not usable with asr_model = "small"; use a larger model and pass language.

Troubleshooting

Development

uv sync --extra dev --extra models
uv run pytest -m "not slow and not long"   # fast suite: fake backends, no downloads
uv run pytest -m slow                      # real models
uv run pytest -m long                      # 30-minute file / async path
uv run python scripts/gen_tool_docs.py --write-examples   # refresh docs/tools.md + examples

Releasing: bump the version in pyproject.toml and src/audio_mcp/__init__.py, add a CHANGELOG.md entry, then push a vX.Y.Z tag. .github/workflows/publish.yml builds the package and uploads it to PyPI through trusted publishing (no API token stored). Release notes: CHANGELOG.md.

Evaluation and benchmarks: docs/evaluation.md. Design spec, implementation plan, and build notes (design rulings and known limits): docs/superpowers/.

Project layout:

src/audio_mcp/
  server.py         MCP entry point (stdio), tool registration, lifespan
  tools/            request schemas and tool handlers
  ingest/           source resolution, path/URL guards, ffmpeg probe and decode
  store/            content-addressed audio store, result cache, retention
  pipeline/         stage DAG, auto planner, runner
  backends/         model adapters (Whisper, PANNs, ECAPA, Silero, MiniLM, LLMs) + fake backends for tests
  dsp/              features, quality, preprocessing, integrity, speech characteristics
  analysis/ timeline/ reasoning/ compare/ library/ stream/ export/ eval/
scripts/            benchmark, evaluate, docs generator
tests/              unit, integration, e2e (stdio)

Known limitations

• Confidence values are raw model probabilities or heuristics, never calibrated.
• Default diarization (ECAPA clustering) does not detect overlapping speech; install the pyannote extra for that.
• Accuracy on low-resource languages with the default small model is poor (see Evaluation).
• Translation is to English only (Whisper).
• Integrity indicators are signal heuristics, not a deepfake or tamper detector.
• Malformed tool arguments are rejected by the MCP layer before the server sees them, so they come back as MCP tool errors rather than the JSON envelope.

Full list: docs/capabilities.md and docs/superpowers/build-notes.md.

License

The code in this repository is released under the Apache License 2.0 (see also NOTICE).

Third-party models and data keep their own licenses, which the Apache License does not cover: faster-whisper (MIT) and Whisper weights (MIT); PANNs code (MIT) and CNN14 weights (CC BY 4.0, trained on AudioSet); SpeechBrain (Apache-2.0) and spkrec-ecapa-voxceleb (Apache-2.0); all-MiniLM-L6-v2 (Apache-2.0); silero-vad (MIT); pyannote.audio (MIT) with gated model terms; AudioSet ontology labels (CC BY 4.0). Test fixtures: speech_es.ogg is a LibriVox public-domain excerpt; the English fixtures are synthesized locally; details in tests/fixtures/audio/SOURCES.md. Check each model's terms before commercial use.