Claude Whisper

Voice-controlled interface for Claude Code using Apple MLX Whisper for speech recognition.

Overview

Claude Whisper enables voice interaction with Claude Code through push-to-talk functionality. Hold a configurable key (default: ESC), speak your wake word followed by a command, and Claude Whisper will transcribe and execute it using the Claude Agent SDK.

Features

• Push-to-talk interface with configurable hotkey
• Real-time speech recognition using MLX Whisper (optimized for Apple Silicon)
• Desktop notifications for task status
• Direct integration with Claude Agent SDK
• Configurable wake word to trigger Claude commands
• TOML-based configuration with environment variable overrides

Requirements

• macOS with Apple Silicon (for MLX acceleration)
• Python 3.10+
• PortAudio (for microphone input)
• Anthropic API key (for Claude Agent SDK)

macOS Permissions

Your terminal application needs the following permissions enabled in System Settings > Security & Privacy:

• Input Monitoring - Required for detecting push-to-talk key presses
• Accessibility - Required for keyboard event monitoring
• Screen Recording - Required for the screenshot tool to capture your screen

Installation

Prerequisites

Install PortAudio (required for microphone input):

brew install portaudio

Using uvx (Recommended)

uvx claude-whisper /path/to/your/project

Using pipx

pipx install claude-whisper
claude-whisper /path/to/your/project

Using pip

pip install claude-whisper
claude-whisper /path/to/your/project

From Source

1. Install system dependencies:

make install-deps

2. Install the package:

uv sync

Usage

Audio Mode (Push-to-Talk)

Run Claude Whisper with a working directory:

claude-whisper /path/to/your/project

Once running:

1. Hold the push-to-talk key (default: ESC)
2. Say your wake word followed by your command (e.g., "Jarvis, create a README for this project")
3. Release the key when done speaking
4. The audio will be transcribed and sent to Claude
5. Desktop notifications will alert you when tasks start and finish

Configuration

Configure Claude Whisper using environment variables with the CLAUDE_WHISPER_ prefix or a TOML config file at ~/.config/claude-whisper/config.toml:

Configuration Options

Example TOML Configuration

Create ~/.config/claude-whisper/config.toml:

model_name = "mlx-community/whisper-medium-mlx-8bit"
command = "jarvis"
push_to_talk_key = "esc"
permission_mode = "acceptEdits"

Available Push-to-Talk Keys

• esc, escape - Escape key
• space - Space bar
• enter - Enter key
• tab - Tab key
• ctrl, shift, alt, cmd - Modifier keys
• Any single character (e.g., a, z, 1)

Development

Testing

Run the test suite:

make test

Run tests with coverage:

pytest --cov=src/claude_whisper --cov-report=term-missing

Run specific test files:

pytest tests/test_config.py
pytest tests/test_main.py
pytest tests/test_integration.py

Code Quality

Format code:

make format

Run linter:

make lint

Fix linting issues:

make lint-fix

Check formatting and linting:

make check

License

See LICENSE file for details.

Author

Ashton Sidhu (ashton@sidhulabs.ca)