sessionvoice (the ears and the mouth)

sessionvoice is the distribution and the Claude Code plugin. The commands and modules keep their names: ears, ears-serve, mouth, src/ears, src/mouth.

One listening engine for VoiceMode and the Delta bot. It is always listening, streams transcription, and decides when he has finished.

PCM (s16le mono) -> backend ----------> events (docs/events.md) -> stdout
                      apple | nemotron | whisper       ^
                    VAD (Silero) ----- edges ----------+
                    turn stage (silence | smart) -- turn, preview

• Backends, all speaking one event format:
  • apple: SpeechTranscriber on macOS 27, through the apple-speech Swift helper.
  • nemotron: Nemotron 3.5 streaming on MLX, run as a child process.
  • whisper: any OpenAI-compatible transcription URL, for Linux or pre-27 macOS.
• One VAD (Silero, falling back to energy). It produces vad edges, and Nemotron segments on them too.
• Turn stages: silence (the bot's join rules, plus commit) and smart (Cora's Smart Turn and Parakeet re-decode), both in-process with the audio.

Components

The engine above is the core. Around it, these parts make it an always-on pair of ears on a Mac (m5, 2026-10-01; depth in docs/architecture/):

• Capture (src/ears/listen): ears.listen capture reads one input device and writes every partial, turn and heartbeat to the heard log (src/ears/heard.py): ~/.voicemode/logs/conversations/heard_YYYY-MM-DD.jsonl.
  • A speaker change ends the turn (--speaker-split K, EARS-14): when another named speaker holds K partials in a row (by the speaker side-car's speakers_YYYY-MM-DD.jsonl), the capture closes the turn before them and opens a new one, so each heard turn has one speaker. A split turn carries speaker (the side-car's label: alice, mike, ...).
  • Every turn the side-car named carries speaker (EARS-106): an unsplit turn is named by the same rule, from its own partials' labels (the one name that held K in a row; only a turn of fewer than K partials, too short to reach K at all, is named by its named labels if they are all one name; a longer turn no name held K in a row, such as a mostly-neutral TV turn with one stray mike, carries none), and marked speaker_from: labels. A turn of neutral labels only (the TV), no labels yet, or a change the split had not cut when the turn closed, carries none. Default 2; 0 = off (no speaker on any turn: today's turns exactly). No side-car file: turns as before. Labels that name nobody (other, mixed, silent, unheard) never count.
  • Deploy turns it on. keeper/ears-keeper.sh starts the capture without --speaker-split, so the default K=2 goes live at the next capture restart after merge. Rollback: add --speaker-split 0 to the capture line in start_capture and restart the capture.
• Keeper (keeper/ears-keeper.sh, launchd com.failmode.ears-keeper): keeps the capture alive on a device that comes and goes. It follows the Mac's default input through keeper/input-watch.swift, and when the AirPods leave or return it stops the capture and starts one on the right device. It only manages the capture a keeper started; other captures (a call-tap, a call leg) are left alone.
• Sounds (keeper/ears-sounds.py, launchd dev.voicemode.ears-sounds): a short sound in his ears when the ears log an event, so he knows he was heard.
• Turn mail (keeper/ears-mail.py, launchd dev.voicemode.ears-mail, EARS-6): mails every finished turn of his To and From heard+<device>@<host>, with the Ears-* headers of the pager genie's contract (sessionmail openspec/changes/pager-genie). A session hears a device by subscribing to that address. The box is heard (--sender; Mike, 16:13 Thu 2026-10-01), because ears is this project's own box. sessionmail trusts only its keeper's From (_voice.KEEPER), so the plist loads only once that says heard too. --to seats is the older mail, To each seat a turn is for. Never run --to seats and --to device for the same seat: it would get each call twice. Ears-Speaker is whose turn it is. A turn with a speaker label (a split turn, EARS-14, or any turn the side-car named, EARS-106) is mailed as <label>@<host>: alice becomes alice@m5, the same form as --speaker. The label is lower-cased, and any character outside [a-z0-9._-] becomes -. Every other turn, including one whose label is empty or names nobody, keeps --speaker ($EARS_SPEAKER, else unknown@<host>: a voice the matcher could not place is never mailed as Mike, EARS-110). Ears-Echo says a mouth line heard back through the mic was cut out of the turn (EARS-98). On the speakers the mic hears the mouth, so a line's words are cut wherever they sit in his turn: at its start (up to 3 + 5 words/s × the seconds the line began after his turn of his words ahead of it), in the middle, or at the end. The aim is to cut only the echo, never his words around it. That is what the fixtures test (8 real echo turns, 8 AirPods quotes), not a guarantee: the cut is a text match. A span starts on 2 or more matched words in a row, or on a lone word joined to them by one-for-one swaps (a misheard name), so a lone "the" of his does not drag his words into a clipped echo. Inside a span, up to 30% unmatched words (half, when the span is a word-for-word copy of the line) are taken as misheard echo. The header is ONE line, a comma list with an entry per line cut, in turn order: Ears-Echo: cora/2d492fb06d3b 31 words cut, cora/6c681f74374e 30 words cut (<seat>/<utt> <n> words cut; the seat is the line's agent). It goes on both the device mail and the --to seats mail; the seats mail also keeps its "heard whole" section. A mail with no cut has no Ears-Echo. A turn that is all echo is still not mailed. Neither is one word left BETWEEN two cuts: that is the seam misheard ("86%" for "eighty-six"). A lone word before or after both cuts is his and is mailed. Timing uses the line's play span (said − played_s .. said) and the turn's real span (the stream start, from the heartbeat or listen-started, + t0..t1), else ts − (t1 − t0). Three guards keep his quotes of a line: a line played on headphones (AirPods, as the mouth's BARGE_DEVICES) is never cut; a cut needs half the line's played words; and, only when the stream start is known (the real span), the turn must have opened within 2 s of the line beginning, unless the turn is mostly the line. On the ts − (t1 − t0) estimate, only the first two guards apply. Numerals (9% for "nine percent") still slip past the word match.
• Waiter (bin/heard-first): an agent blocks on the heard log until a line is for it, by --rule: its name in a partial or a turn (partial:REGEX, turn:REGEX), a reply to its last line (reply:AGENT), or his first words (first); then it wakes. bin/ears-turn then reads the whole turn, one line.
• Mouth (src/mouth, bin/mouth): an agent speaks through one queue, and its lines go into the same log, so the waiters can gate out echo.
• Calls: bin/call-tap puts a call from any app into the ears, both ways; keeper/call-ears grows an ear for each Delta call and drops it at hang-up. The ears keeper runs it as its child: one launch agent, com.voicemode.ears-keeper (ears services install), holds the room's ear and every call's.
• Speaker names (not in this repo yet): a sidecar tags each heard line with who spoke, in speakers_YYYY-MM-DD.jsonl beside the heard log, and the waiters read it. Moving it into VoiceMode as an opt-in service is VM-2326.

Use

ears backends                          # what runs here
ears install                           # fetch Silero (the engine: Parakeet on mlx-audio :8890)
ears install --apple                   # also build the Apple helper (swift, macOS 27) + assets
ears file talk.wav --realtime --turns silence --previews
rec ... | ears listen --turns silence  # live
ears serve 2023                        # a whisper-compatible URL on Apple STT

In Python:

from ears.cli import Pipeline
p = Pipeline("auto", turns="silence", previews=True, emit=my_handler)
p.feed(pcm_bytes)   # every 20 ms
p.close()

Install

The distribution is sessionvoice (sessionvoice[nemotron], sessionvoice[all]); it installs the ears, ears-serve and mouth commands.

uv venv && uv pip install -e '.[all]'  # onnxruntime + mlx-audio
ears install

Tests

One command runs the whole suite, offline:

make test                  # uv run --locked pytest
make test ARGS="-k mouth"  # pytest flags pass through

uv sync (or make sync) once installs the dev group from uv.lock; after that nothing in the suite touches the network, and nothing plays a sound. Without sounddevice (PortAudio) the four tests/test_ears_sounds_call.py tests skip: uv sync --extra capture brings it (and onnxruntime, for the silero test), and then they run, and three of them play a short tick through the default output. Without VoiceMode on PYTHONPATH its three tests skip.

Measured (m5, 2026-09-19; Cora's eval set, her score.py, whisper normaliser)

WER through the ears pipeline:

• The real-time call is the 20 turns joined with 3 s gaps, using silence turns with commits on. It produced 22 turns for 20 inputs.
• The old helper, which finalised the live analyzer at each commit, scored 5.17% on the same call, with 54 deletions.
• Turn latency on turn01, fed in real time:
  • Apple fires 1.82 s after speech ends, including a 0.2 s commit wait.
  • Nemotron fires at 1.65 s.
  • VAD edges lag about 140 ms at onset and about 280 ms at offset (a 0.2 s offset debounce).
• Memory: Apple uses about 75 MB of RSS on our side; Nemotron about 2.2 GB.
• Turn stages (Cora, 20 turns, real time): silence cut 1 of 20, with a p50 of 1.59 s. smart veto cut 0 of 20, with a p50 of 2.43 s.

The mouth (src/mouth)

The speaking half: mouth say queues a line (a maildir), a resident player speaks it through Kokoro or a cloned voice, to the room or into a Delta call, and writes saying/said rows to the heard log so the waiters can tell echo from speech. Design: src/mouth/DESIGN.md.

mouth say --voice pip --device default "Hello."
mouth say --voice pip --device call --hold turn-end "On the call."
mouth --help            # stop, retract, amend, status, devices, serve

Moved here from VoiceMode feat/mouth-say @ 64cccd80 (Mike, 00:43 Mon 2026-09-28: "we put the mouth with the ears"). The live player still runs the VoiceMode copy until cutover. Three ties to VoiceMode remain, and they come over with the capture:

• voice_mode.heard: the heard log's reader and writer, shared with the capture.
• voice_mode.voice_profiles: resolves cloned voices (pip, laurie).
• voice_mode.dj: optional, loaded only for music ducking.

Tests: make test ARGS=tests/mouth (see Tests above); put a VoiceMode checkout on PYTHONPATH to run the three tests that need it.