Vybespk

Talk to Claude Code or Codex hands-free while you do something else: washing dishes, driving, tidying up. Dictate a prompt in your own words, with pauses as long as you need, ask side questions and get spoken answers, approve permission prompts by voice, and hear a short summary when the agent finishes. Nothing is sent until you say so.

macOS only for now, with earbuds or a headset with a microphone (AirPods work). Through the laptop's own speakers it hears itself, so earbuds are needed for now. Speech recognition, the voice, and speaker recognition run on your Mac. Drafting, summaries, and research use Claude through Vybespk, so no API keys are needed: new accounts get $2 of credits to try it (about 5 hours), then Pro or a credit pack. Prefer your own Anthropic or OpenAI keys? That's the Own Keys plan.

Install

npx vybespk setup        # or: uv tool install vybespk / pipx install vybespk

The coding agent runs inside tmux; if it isn't installed, setup offers to install it with Homebrew (brew install tmux).

setup signs you in with a code sent to your email (new accounts get $2 of credits), asks how to pay for the AI calls (keep using your credits, Pro, a credit pack, or Own Keys; a purchase opens Stripe's checkout and this Mac picks it up when you've paid), and which coding agent to drive (Claude Code or Codex). With Own Keys it also asks which LLM (Anthropic or OpenAI) and decision model to use and saves your API keys (readable only by you). It downloads the local models (about 700 MB, once) and checks everything works. Then, in any project folder:

vybespk                  # start the agent here and start listening

You need Claude Code (npm i -g @anthropic-ai/claude-code) or Codex (npm i -g @openai/codex) installed. See vybespk --help for every command.

Development

Run from a checkout with ./voicectl (the same commands as vybespk):

brew install tmux
python3 -m venv .venv
.venv/bin/pip install -e .          # vybespk and its dependencies
export ANTHROPIC_API_KEY=...        # drafting, summaries, research
export JEV_API_KEY=...              # routing each utterance (Typesafe JEV)

Settings live in ~/.config/vybespk/config.toml (only what you change) on top of voice_agent/default_config.toml; state and models in ~/Library/Application Support/Vybespk. Releasing: see RELEASE.md. Below, "config.toml" means your settings file; vybespk config prints every setting with its explanation.

Use

vybespk start ~/path/to/project      # Claude Code in tmux, with voice hooks attached
vybespk send "add a config loader"   # type a prompt and press Enter
vybespk answer yes                   # answer an open permission dialog
vybespk interrupt                    # interrupt the current turn
vybespk resume [id | "name" | --list]  # go back to a Claude Code conversation, by ID or /rename name
vybespk stop                         # end the Claude Code session (resume later with claude --resume)
vybespk restart ~/path/to/other      # switch Claude Code to another folder
vybespk state                        # ready, busy, dialog, trust, or starting
vybespk attach                       # watch or type in the session yourself

Extra Claude Code flags go after --, for example vybespk start . -- --permission-mode default.

Hooks are attached only to sessions started by vybespk, through claude --settings. Your normal Claude Code sessions are unaffected.

One command

./voicectl link     # once: installs a `voice` command
cd ~/Documents/repo/writer
voice               # starts Claude Code in this folder (Opus 5.5, accept-edits mode) and listens

voice always uses the folder you run it in. If Claude Code is already running there, it reuses it; if it's idle in another folder, it moves it here (the old conversation stays resumable). It won't move a Claude Code that is busy or waiting on a permission.

Talking to it

Use headphones or earbuds. Through laptop speakers the microphone hears the assistant's own voice, interrupts it, and transcribes it as your speech.

vybespk start ~/Documents/repo/myproject -- --permission-mode manual   # terminal 1
vybespk listen                                                          # terminal 2

Then just talk, or type in the box at the bottom of the window and press Enter: typed text goes the same way as speech, so dictation, questions, "command, ..." and "send it" all work. The window shows the draft, anything held, what Claude Code is doing right now (working or waiting, for how long, its current step, and its last result), and the last few things the assistant said. vybespk watch shows Claude Code's own screen, read-only, in another terminal.

Say	What happens
Describe the work	Builds the draft. A soft pop means the draft changed.
"Actually, make it X"	Rewrites that part of the draft.
Talk to someone else	Dropped, with a bottle sound. Kept until you submit, in case it was for the assistant.
Something ambiguous	Held, with a tink, and asked about when you get back on topic, but only if it's worth asking: the better speech model (small.en) must be confident in it, it must be at least four words, and JEV must judge it related to the current topic. Garbled or unrelated scraps are dropped instead, and "that was me" can still recover them.
Ask it something: "what's the best way to...?"	Answered out loud, with detail on screen. Never added to the draft.
A follow-up: "isn't that heavy?"	Continues the same side discussion.
"Use that" / "add that to the prompt"	Merges the latest answer into the draft.
"Can you hear me?" / "What directory are we in?"	Answers right away from session facts: folder, branch, Claude Code state.
"That was a question for you"	Takes it back out of the draft, held items, or dropped speech, and answers it.
"No, that was for the prompt" / "I was talking to my son"	Moves the earlier utterance where you meant it to go.
"Send it"	Spoken summary of the important parts, then "Shall I send it?" Saying what to send in the same breath ("just put continue in and send that") fills the draft first.
"Yes"	Typed into Claude Code. A glass sound confirms. "Yes?", "yeah", "go ahead" all count; "no" or "not yet" cancels.
"Read it back to me"	The exact prompt text.
"Where were we?"	Topic, draft, held items, last result.
"New topic" / "Start over" / "Scratch that"	As they say.
"Stop"	Interrupts Claude Code's current turn.
A long pause mid-sentence	A sentence that plainly stopped partway ("I'd like you to correct the...") waits up to 2 minutes for the rest, shown on screen as WAITING FOR THE REST. When you speak again, JEV decides whether you're finishing it or starting something new; a command or a new topic leaves it standing alone.
A pause mid-sentence	Anything longer than a command waits 1.5 seconds in case you continue. If you do, the pieces' audio is joined and transcribed once, so Whisper hears the whole sentence. Commands and answers like "send it" or "yes" act at once.
Long speech vs. commands	Two speech models: base.en for every utterance and for commands (about 1 s), small.en for dictation and longer questions once the sentence is complete (about 3 s on a 2014 Intel Mac, much less on Apple Silicon). Far-away speech: 12% of words wrong with base.en alone, 5% with both. Set model and accurate_model under [stt].
"Yes" / "No" to a permission	Answers the dialog, with "Allowed." or "Denied." Destructive actions (deleting, pushing, installing, writing outside the project, sending data out) need "yes, I'm sure". A "yes" said to someone else approves nothing.
Answering a question it asked	It confirms with "Got it."

Talking over the assistant cuts it off. What happens next depends on why you cut in:

Interruption	Result
Noise, or a sound with no words	It resumes on its own once the room is quiet.
Talking to someone else, or "hang on"	It resumes after you go quiet, starting with "As I was saying".
"Go on" / "what were you saying?"	It resumes right away.
Anything else: a question, dictation, a correction	You moved on, so the interrupted reply is dropped.

It replays from the start of the sentence you cut off. Set resume_after_interrupt to always or never in config.toml to change this. vybespk barge-ins shows recent interruptions and how each was handled. vybespk noise-test 30 listens to the room for 30 seconds and counts how often noise, such as a dishwasher, running water, or the TV, would have cut the assistant off. The session is saved to session.json in the state folder and survives a restart. vybespk reset clears it.

Testing the router

.venv/bin/python -m evals.run_routing     # routing and correction cases
.venv/bin/python -m evals.noise_bargein   # synthetic household noise against barge-in
.venv/bin/python -m evals.interrupt_replay  # interruptions: resume, hold, drop, races
.venv/bin/python -m evals.command_replay    # the command keyword and spoken settings
.venv/bin/python -m evals.fragment_eval     # far-away speech in pieces: joined as audio vs not, live JEV calls, under a cent

Commands

Say "command" and then a short phrase, and it's treated as a command, never as dictation. Say "command" alone, or ask "what commands are available?" or "which menu options do I have?", and it lists them. If you pause after "command", it waits a moment for the rest.

Command	What it does
"command, calibrate my voice"	Learns your voice on the current microphone. It reads each sentence aloud and you repeat it; "skip" skips one, "stop" cancels.
"command, learn my wife's voice"	The same for someone else, so the filter can tell you apart.
"command, pause settings"	Reads the current pauses.
"command, answer wait two seconds"	Changes a pause. Any pause works: end of speech, continuation, answer wait, breath gap, resume wait, barge in, progress, permission reminder. Say a number of seconds, or "longer" or "shorter".
"command, status updates off" / "on" / "every 30 seconds"	Turns the spoken status updates off or on, or changes how often they come. Setting an interval turns them on. Permission reminders play either way.
"command, say one sec after one second"	How long it waits before saying "one sec" while an answer is coming.
"command, which microphone"	Says which microphone it's listening through and why, and whether your earbuds are in music or headset mode.
"command, use the laptop microphone" / "use the earbuds microphone" / "choose the microphone automatically"	Switches microphones for this session. Choosing the Sony earbuds' mic warns that it puts them in headset mode.
"command, permission mode"	Says the current mode, lists the six modes, and asks which you want. "Command, permission mode auto" switches directly. Manual, accept edits, plan, and auto switch in place; don't ask and bypass restart the same conversation. Bypass needs "I'm sure". Your choice becomes the default. Same as voice mode auto.
"mute" / "unmute"	Said on its own (or after "command"). While muted, Vybespk acts on nothing it hears or is typed until you say "unmute", and only your voice unmutes it. Handy for a phone call. "The mute button should be red" is still dictation.
"command, resume the last session"	Restarts Claude Code on the conversation before the current one, in that conversation's folder. Refused while Claude Code is busy. Same as vybespk resume; vybespk resume --list shows recent and named ones.
"command, resume the voice agent session"	The same, by the name you gave a session with /rename. Same as vybespk resume "voice agent".
"command, send it", "read it back", "where were we", "scratch that", "start over", "new topic", "stop"	The usual controls, without any chance of being taken as dictation.

The same requests also work without the keyword, such as "wait two seconds before you answer", when they're clear. Changes are saved to your settings file (~/.config/vybespk/config.toml).

Recognizing your voice

Once you enroll, only your voice reaches the prompt, and only your voice can interrupt the assistant. Your wife, your son, and the TV are dropped, and that speech is still logged, so "that was me" brings it back.

vybespk enroll            # read ten sentences, about a minute; do it once per microphone
                             # (or, while listening, say "command, calibrate my voice")
vybespk enroll wife       # optional, a few sentences each; makes the filter much sharper
vybespk enroll son
vybespk whois 30          # talk, have others talk, and see what it decides
vybespk enroll --list     # profiles, pitch, and your threshold

It uses a WeSpeaker voice model through sherpa-onnx (about 75 ms per utterance) plus pitch: a voice much higher than yours counts as someone else. Each utterance is judged "me", "other", or "unsure"; unsure speech goes through as before. A barge-in is checked after 0.6 s of speech, and again at 1.2 s if unsure; doubtful cases lean toward letting you in. Tune [speaker_id] in config.toml if needed.

Voice

Speech uses a local Piper neural voice through sherpa-onnx, with macOS say as the fallback. Nothing leaves the Mac.

.venv/bin/pip install --only-binary=:all: "sherpa-onnx==1.13.8"   # 2.x drops Piper's pronouncer
vybespk voices                          # list voices, * marks the current one
vybespk voice-test en_US-lessac-high    # hear one (downloads it if needed)
vybespk voice en_US-lessac-high         # switch, then restart listen

Voices live in the models folder (~/Library/Application Support/Vybespk/models). Measured on a 2014 Intel MacBook Pro, "medium" voices generate about 17 times faster than real time and "high" voices about 3 times, so "high" voices start each reply roughly a second later. While AirPods or other Bluetooth earbuds have their microphone in use, playback drops to phone-call quality whichever voice you pick; a separate microphone keeps it at full quality.

Some earbuds sound distorted in that headset mode. On this Mac the Sony WF-1000XM5 are clear in music mode and garbled in headset mode, so they're listed in output_only_devices in config.toml: while they're the output, the listener hears through the laptop microphone and they stay in music mode. AirPods use their own microphone as usual. Voice recognition is per microphone, so calibrate once on the laptop mic too ("command, calibrate my voice").

Models

Work	Model
Routing every utterance	JEV (jev-1.14.0)
Drafting, quick replies, spoken summaries	Claude Haiku 5.5
Side questions: searches, quick lookups, basic coding questions	Claude Sonnet 5.5 (low effort for simple lookups)
Complex questions: code analysis, debugging, design, "dig into this"	Claude Opus 5.5
Claude Code, for every prompt you submit	Claude Opus 5.5 or stronger

vybespk start always launches Claude Code with --model claude-opus-5-5 (set in [claude_code] in config.toml; Fable 5.1 and Mythos 5.1 are also allowed). Before each voice prompt is submitted, the bridge checks the model on screen. If it's weaker, it restarts Claude Code on Opus 5.5 with --resume, keeping the conversation, and then sends. It never uses /model, because that also changes your default for every new session.

Questions never stop Claude Code. Only an explicit "stop", "cancel", or "abort" does.

Claude Code starts in acceptEdits mode by default (permission_mode in config.toml), so file reads and edits in the project don't ask; other shell commands still do. A permission prompt is never dropped: if it's cut off or goes unanswered, it's asked again when you're quiet, and every 30 seconds it reminds you that Claude Code is waiting.

Setup: coding agent, LLM, and decision model

Three settings, each its own choice in config.toml:

Setting	Options	What it is
[agent] name	claude_code, codex	The coding agent the voice agent drives in tmux
[llm] provider	anthropic, openai	The LLM behind drafting, summaries, quick replies, research
[llm] decisions	jev, openai	The decision model that routes every utterance

vybespk setup              # ask each question, save the keys each choice needs, test them
vybespk setup openai       # everything on OpenAI: Codex + GPT models + Decisions API
vybespk setup anthropic    # Claude Code + Claude models + JEV
vybespk setup show         # what's chosen, and where each key comes from

Model tiers are per provider ([llm.anthropic], [llm.openai]): fast for drafting and summaries (Haiku 5.5 / GPT-6 Luna), research (Sonnet 5.5 / GPT-6.1 Sol), and deep (Opus 5.5 / GPT-6 Astra). Codex runs on gpt-6.1-sol or stronger ([codex]), as Claude Code runs on Opus 5.5 or stronger. Adding a provider or agent is one module: voice_agent/llm/<name>_provider.py or voice_agent/agents/<name>.py.

Codex permission modes, by voice ("command, permission mode") or vybespk mode: read only, workspace (the default: edits and runs commands in the project, asks about anything outside it), auto review, full access, and no approvals (needs "I'm sure"). Every switch restarts the same conversation. The first time Codex starts with the voice hooks it asks you to trust them; choose "Review hooks" if you have other hooks of your own, or "Trust all and continue". It asks again only if the hooks change.

Keys are saved in ~/.config/voice/keys.json, readable only by you; a key set in your shell takes precedence. These choices apply with Own Keys; otherwise the AI calls go through Vybespk with Claude and JEV and no keys are needed.

Plans and usage

Everyone signs in (vybespk login): a 6-digit code is emailed, and the same address signs you in on up to 3 Macs.

Plan	How it's paid
Trial	$2 of credits when you sign up, about 5 hours of use. No API keys needed.
Credits	Prepaid packs ($10, $20). Each call comes off your balance at twice its API cost.
Pro, $12/month	Everything included except research, which gets 60 questions a month; past that, research uses credits. Includes Own Keys.
Own Keys, $5/month	Your own Anthropic or OpenAI (and JEV) keys pay for the AI calls, which go straight to the provider.

vybespk usage [days]       # every model call on this machine and what it cost, on any plan
vybespk buy                # pick a plan; checkout opens in your browser, this Mac activates when paid
vybespk plan               # your plan, credits left, research questions used
vybespk billing            # cancel Pro, change your card, see receipts and invoices
vybespk login              # sign in on this Mac (or another one) with a code by email
vybespk logout             # sign this Mac out; frees its slot

vybespk share on           # opt in to sharing your data; earns $2 of credits a month
vybespk share status       # what's waiting to upload, progress toward this month's reward
vybespk share off          # stop, and delete anything not yet uploaded

Sharing collects each utterance's audio and transcripts, how it was handled, the prompts you send, and research answers. It never includes speech the voice filter says is someone else's, or anything that looks like a password or key. The reward is $2 of credits for a month with at least 200 utterances on 3 or more days. A license works on up to 3 machines; vybespk deactivate frees one.

On a plan, the voice agent's own models are Claude and JEV through our relay at api.vybespk.com, whatever LLM setting you chose. The coding agent still runs on your own account: Codex on your OpenAI or ChatGPT sign-in, Claude Code on your Claude one.

Payments go through Stripe, which is the seller of record: it handles sales tax and VAT and sends receipts. Paid plans go through the relay in relay/ (see relay/README.md to deploy it). When a paid plan runs out, the assistant says so ("You're out of research questions this month") and keeps doing whatever's still covered.

Progress while you wait

Status updates yield to you: the moment you start talking, an update stops mid-sentence (no voice check, since it's low-stakes), and once you've been quiet for a few seconds a fresh one plays with the current status. "Command, update quiet two seconds" changes how long it waits.

When a question is accepted you hear a soft tone. If the answer takes more than 5 seconds it says "One sec." During long research or a long Claude Code task, it says a few words about what's happening about every 12 seconds, such as "Claude Code is running the unit tests" or "Checking how the speaker picks the next item." Updates never play while you're talking. Claude Code updates come from hooks, so restart a session started before this change to get them.

What the assistant can and can't do itself

It reads: project files, read-only git (status, log, diff, show, branch, blame), and listing commands (ls, find, du) in your home folder. Anything that changes something, such as edits, tests, builds, commits, or installs, goes to Claude Code through the draft, after you confirm.

What the hooks do

Hook	Behavior
Stop	Summarizes Claude Code's final response with Claude Haiku 5.5 and speaks it. Short plain replies are spoken as they are.
PermissionRequest	Speaks what Claude Code wants to do, such as "Claude wants to write todo.txt. Allow it?"
Notification	Speaks elicitation and needs-input notifications.

All hooks run async, so they never block Claude Code. Spoken text is logged to spoken.log in the state folder.

Spike findings

• Bracketed paste through tmux paste-buffer -p delivers multi-line prompts as one message. Embedded newlines do not submit early.
• Keys sent while Claude Code is starting are silently dropped. The bridge now waits for the input box before typing.
• The folder trust dialog defaults to "No, exit". The bridge reports it rather than accepting it, so trusting a folder stays a human decision.
• The Stop hook receives the final response text directly. Summary latency is about 0.7 to 1 second on Claude Haiku 5.5. Most of the wait you hear is speech playback.
• The summary is much better when it knows what was asked. The bridge records each prompt it sends and passes it to the summarizer.
• Not every tool call raises a permission dialog. Answer keys are sent only when a dialog is actually on screen, otherwise they would land in the prompt box as text.
• Escape interrupts a running turn. The Stop hook does not fire on interrupt.
• Your default permission mode is auto, which approves most actions without a dialog. Use --permission-mode default to hear permission prompts.

Known gaps for later phases

• Pane state is detected by matching on-screen text, which can break when Claude Code's TUI changes. A hook-driven state file would be sturdier.
• No destructive-action phrase check yet. "Yes" approves any dialog.
• Prompts typed by hand in the session are not recorded, so their summaries lack the request context. The transcript path in the hook input can fix this.
• Background task completions produce extra spoken summaries.

License

Functional Source License 1.1, Apache 2.0 future license (FSL-1.1-ALv2): use, change and share it for anything except a competing product; each version becomes Apache 2.0 two years after release. faster-whisper, vendored in voice_agent/_vendor/, is MIT.