wyoming 1.10.0

Peer-to-peer protocol for voice assistants

Wyoming Protocol

A peer-to-peer TCP protocol for voice assistants (basically JSONL + PCM audio)

{ "type": "...", "data": { ... }, "data_length": ..., "payload_length": ... }\n
<data_length bytes (optional)>
<payload_length bytes (optional)>

Used in Home Assistant for communication with voice services.

Wyoming Projects

• Audio input/output
  • mic-external
  • snd-external
  • SDL2
• Wake word detection
  • openWakeWord
  • porcupine1
  • snowboy
  • microWakeWord
• Speech-to-text
  • Faster Whisper
  • Vosk
  • Whisper.cpp
• Text-to-speech
  • Piper
• Intent handling
  • handle-external

Format

1. A JSON object header as a single line with \n (UTF-8, required)
   • type - event type (string, required)
   • data - event data (object, optional)
   • data_length - bytes of additional data (int, optional)
   • payload_length - bytes of binary payload (int, optional)
2. Additional data (UTF-8, optional)
   • JSON object with additional event-specific data
   • Merged on top of header data
   • Exactly data_length bytes long
   • Immediately follows header \n
3. Payload
   • Typically PCM audio but can be any binary data
   • Exactly payload_length bytes long
   • Immediately follows additional data or header \n if no additional data

Event Types

Available events with type and fields.

Audio

Send raw audio and indicate begin/end of audio streams.

• audio-chunk - chunk of raw PCM audio
  • rate - sample rate in hertz (int, required)
  • width - sample width in bytes (int, required)
  • channels - number of channels (int, required)
  • timestamp - timestamp of audio chunk in milliseconds (int, optional)
  • Payload is raw PCM audio samples
• audio-start - start of an audio stream
  • rate - sample rate in hertz (int, required)
  • width - sample width in bytes (int, required)
  • channels - number of channels (int, required)
  • timestamp - timestamp in milliseconds (int, optional)
• audio-stop - end of an audio stream
  • timestamp - timestamp in milliseconds (int, optional)

Info

Describe available services.

• describe - request for available voice services
• info - response describing available voice services
  • asr - list speech recognition services (optional)
    • models - list of available models (required)
      • name - unique name (required)
      • languages - supported languages by model (list of string, required)
      • attribution (required)
        • name - name of creator (required)
        • url - URL of creator (required)
      • installed - true if currently installed (bool, required)
      • description - human-readable description (string, optional)
      • version - version of the model (string, optional)
    • supports_transcript_streaming - true if program can stream transcript chunks
    • requires_external_vad - true if ASR program needs external VAD to detect end of voice commands
    • prefers_auto_gain_enabled - true if input audio should have auto gain enabled
    • prefers_noise_reduction_enabled - true if input audio should have noise reduction enabled
  • tts - list text to speech services (optional)
    • models - list of available models
      • name - unique name (required)
      • languages - supported languages by model (list of string, required)
      • speakers - list of speakers (optional)
        • name - unique name of speaker (required)
      • attribution (required)
        • name - name of creator (required)
        • url - URL of creator (required)
      • installed - true if currently installed (bool, required)
      • description - human-readable description (string, optional)
      • version - version of the model (string, optional)
    • supports_synthesize_streaming - true if program can stream text chunks
  • wake - list wake word detection services (optional)
    • models - list of available models (required)
      • name - unique name (required)
      • languages - supported languages by model (list of string, required)
      • attribution (required)
        • name - name of creator (required)
        • url - URL of creator (required)
      • installed - true if currently installed (bool, required)
      • description - human-readable description (string, optional)
      • version - version of the model (string, optional)
  • handle - list intent handling services (optional)
    • models - list of available models (required)
      • name - unique name (required)
      • languages - supported languages by model (list of string, required)
      • attribution (required)
        • name - name of creator (required)
        • url - URL of creator (required)
      • installed - true if currently installed (bool, required)
      • description - human-readable description (string, optional)
      • version - version of the model (string, optional)
    • supports_handled_streaming - true if program can stream response chunks
    • supports_home_control - true if program may execute home control actions during handling
  • intent - list intent recognition services (optional)
    • models - list of available models (required)
      • name - unique name (required)
      • languages - supported languages by model (list of string, required)
      • attribution (required)
        • name - name of creator (required)
        • url - URL of creator (required)
      • installed - true if currently installed (bool, required)
      • description - human-readable description (string, optional)
      • version - version of the model (string, optional)
  • satellite - information about voice satellite (optional)
    • area - name of area where satellite is located (string, optional)
    • has_vad - true if the end of voice commands will be detected locally (boolean, optional)
    • active_wake_words - list of wake words that are actively being listened for (list of string, optional)
    • max_active_wake_words - maximum number of local wake words that can be run simultaneously (number, optional)
    • supports_trigger - true if satellite supports remotely-triggered pipelines
  • mic - list of audio input services (optional)
    • mic_format - audio input format (required)
      • rate - sample rate in hertz (int, required)
      • width - sample width in bytes (int, required)
      • channels - number of channels (int, required)
  • snd - list of audio output services (optional)
    • snd_format - audio output format (required)
      • rate - sample rate in hertz (int, required)
      • width - sample width in bytes (int, required)
      • channels - number of channels (int, required)
• select-program - selects which program handles the connection (optional)
  • name - name of the program to use, matching a program name from info (string, required)
  • Sent after connecting, before the first request event (e.g. describe, transcribe, synthesize, detect, recognize)
  • Applies for the lifetime of the connection
  • The domain (asr, tts, ...) is implied by the request events that follow, so name only needs to be unique within a domain
  • If not sent, the first program of each type in info is used
  • Servers are expected to drop unrecognized events, so sending this to a server that predates it is a no-op (the default program is used)

Speech Recognition

Transcribe audio into text.

• transcribe - request to transcribe an audio stream
  • name - name of model to use (string, optional)
  • language - language of spoken audio (string, optional)
  • context - context from previous interactions (object, optional)
  • vad_sensitivity - how quickly end of voice command is detected (string, optional)
    • Only if ASR program has requires_external_vad as False
• transcript - response with transcription
  • text - text transcription of spoken audio (string, required)
  • language - language of transcript (string, optional)
  • context - context for next interaction (object, optional)

Streaming:

1. transcript-start - starts stream
   • language - language of transcript (string, optional)
   • context - context from previous interactions (object, optional)
2. transcript-chunk
   • text - part of transcript (string, required)
3. Original transcript event must be sent for backwards compatibility
4. transcript-stop - end of stream

Text to Speech

Synthesize audio from text.

• synthesize - request to generate audio from text
  • text - text to speak (string, required)
  • voice - use a specific voice (optional)
    • name - name of voice (string, optional)
    • language - language of voice (string, optional)
    • speaker - speaker of voice (string, optional)
  • text_format - format of text (string, optional)
    • Can be "text", "ssml", or something else

Streaming:

1. synthesize-start - starts stream
   • context - context from previous interactions (object, optional)
   • voice - use a specific voice (optional)
     • name - name of voice (string, optional)
     • language - language of voice (string, optional)
     • speaker - speaker of voice (string, optional)
   • text_format - format of text (string, optional)
     • Can be "text", "ssml", or something else
2. synthesize-chunk
   • text - part of text to synthesize (string, required)
3. Original synthesize message must be sent for backwards compatibility
4. synthesize-stop - end of stream, final audio must be sent
5. synthesize-stopped - sent back to server after final audio

Wake Word

Detect wake words in an audio stream.

• detect - request detection of specific wake word(s)
  • names - wake word names to detect (list of string, optional)
• detection - response when detection occurs
  • name - name of wake word that was detected (string, optional)
  • timestamp - timestamp of audio chunk in milliseconds when detection occurred (int, optional)
• not-detected - response when audio stream ends without a detection

Voice Activity Detection

Detects speech and silence in an audio stream.

• voice-started - user has started speaking
  • timestamp - timestamp of audio chunk when speaking started in milliseconds (int, optional)
• voice-stopped - user has stopped speaking
  • timestamp - timestamp of audio chunk when speaking stopped in milliseconds (int, optional)

Intent Recognition

Recognizes intents from text.

• recognize - request to recognize an intent from text
  • text - text to recognize (string, required)
  • context - context from previous interactions (object, optional)
• intent - response with recognized intent
  • name - name of intent (string, required)
  • entities - list of entities (optional)
    • name - name of entity (string, required)
    • value - value of entity (any, optional)
  • text - response for user (string, optional)
  • context - context for next interactions (object, optional)
• not-recognized - response indicating no intent was recognized
  • text - response for user (string, optional)
  • context - context for next interactions (object, optional)

Multiple intents:

1. intents-start - signals one or more intents will follow
   • context - context from previous interactions (object, optional)
2. intent - one or more intents sent as normal
   • Older clients will only process the first intent
3. intents-stop - end of intents

Intent Handling

Handle structured intents or text directly.

• handled - response when intent was successfully handled
  • text - response for user (string, optional)
  • context - context for next interactions (object, optional)
• not-handled - response when intent was not handled
  • text - response for user (string, optional)
  • context - context for next interactions (object, optional)

Streaming:

1. handled-start - starts stream
   • context - context from previous interactions (object, optional)
2. handled-chunk
   • text - part of response (string, required)
3. Original handled message must be sent for backwards compatibility
4. handled-stop - end of stream

Audio Output

Play audio stream.

• played - response when audio finishes playing

Voice Satellite

Control of one or more remote voice satellites connected to a central server.

• run-satellite - informs satellite that server is ready to run pipelines
• pause-satellite - informs satellite that server is not ready anymore to run pipelines
• satellite-connected - satellite has connected to the server
• satellite-disconnected - satellite has been disconnected from the server
• streaming-started - satellite has started streaming audio to the server
• streaming-stopped - satellite has stopped streaming audio to the server

Pipelines are run on the server, but can be triggered remotely from the server as well.

• run-pipeline - runs a pipeline on the server or asks the satellite to run it when possible
  • start_stage - pipeline stage to start at (string, required)
  • end_stage - pipeline stage to end at (string, required)
  • wake_word_name - name of detected wake word that started this pipeline (string, optional)
    • From client only
  • wake_word_names - names of wake words to listen for (list of string, optional)
    • From server only
    • start_stage must be "wake"
  • announce_text - text to speak on the satellite
    • From server only
    • start_stage must be "tts"
  • restart_on_end - true if the server should re-run the pipeline after it ends (boolean, default is false)
    • Only used for always-on streaming satellites

Timers

• timer-started - a new timer has started
  • id - unique id of timer (string, required)
  • total_seconds - number of seconds the timer should run for (int, required)
  • name - user-provided name for timer (string, optional)
  • start_hours - hours the timer should run for as spoken by user (int, optional)
  • start_minutes - minutes the timer should run for as spoken by user (int, optional)
  • start_seconds - seconds the timer should run for as spoken by user (int, optional)
  • command - optional command that the server will execute when the timer is finished
    • text - text of command to execute (string, required)
    • language - language of the command (string, optional)
• timer-updated - timer has been paused/resumed or time has been added/removed
  • id - unique id of timer (string, required)
  • is_active - true if timer is running, false if paused (bool, required)
  • total_seconds - number of seconds that the timer should run for now (int, required)
• timer-cancelled - timer was cancelled
  • id - unique id of timer (string, required)
• timer-finished - timer finished without being cancelled
  • id - unique id of timer (string, required)

Miscellaneous

• user-event - user-defined event
  • name - name of the user event type (string, required)
  • data - data for user event (object, optional)
  • context - context from previous interactions (object, optional)

Event Flow

• → is an event from client to server
• ← is an event from server to client

Service Description

1. → describe (required)
2. ← info (required)

Program Selection

When an endpoint exposes more than one program of a type (e.g. multiple asr programs), select one for the connection before the request events below:

1. → select-program with name of the program to use (optional)
   • Omit to use the first program of each type in info
2. → request events for the chosen program (e.g. transcribe, synthesize, detect, recognize)

Speech to Text

1. → transcribe event with name of model to use or language (optional)
2. → audio-start (required)
3. → audio-chunk (required)
   • Send audio chunks until silence is detected
4. → audio-stop (required)
5. ← transcript (required)
   • Contains text transcription of spoken audio

Streaming:

1. → transcribe event (optional)
2. → audio-start (required)
3. → audio-chunk (required)
   • Send audio chunks until silence is detected
4. ← transcript-start (required)
5. ← transcript-chunk (required)
   • Send transcript chunks as they're produced
6. → audio-stop (required)
7. ← transcript (required)
   • Sent for backwards compatibility
8. ← transcript-stop (required)

Text to Speech

1. → synthesize event with text (required)
2. ← audio-start
3. ← audio-chunk
   • One or more audio chunks
4. ← audio-stop

Streaming:

1. → synthesize-start event (required)
2. → synthesize-chunk event (required)
   • Text chunks are sent as they're produced
3. ← audio-start, audio-chunk (one or more), audio-stop
   • Audio chunks are sent as they're produced with start/stop
4. → synthesize event
   • Sent for backwards compatibility
5. → synthesize-stop event
   • End of text stream
6. ← Final audio must be sent
   • audio-start, audio-chunk (one or more), audio-stop
7. ← synthesize-stopped
   • Tells server that final audio has been sent

Wake Word Detection

1. → detect event with names of wake words to detect (optional)
2. → audio-start (required)
3. → audio-chunk (required)
   • Keep sending audio chunks until a detection is received
4. ← detection
   • Sent for each wake word detection
5. → audio-stop (optional)
   • Manually end audio stream
6. ← not-detected
   • Sent after audio-stop if no detections occurred

Voice Activity Detection

1. → audio-chunk (required)
   • Send audio chunks until silence is detected
2. ← voice-started
   • When speech starts
3. ← voice-stopped
   • When speech stops

Intent Recognition

1. → recognize (required)
2. ← intent if successful
3. ← not-recognized if not successful

For multiple intents:

1. → recognize (required)
2. ← intents-start if successful
3. ← intent if successful
   • One or more intents
4. ← intents-stop if successful
5. ← not-recognized if not successful

Intent Handling

For structured intents:

1. → intent (required)
2. ← handled if successful
3. ← not-handled if not successful

For text only:

1. → transcript with text to handle (required)
2. ← handled if successful
3. ← not-handled if not successful

Streaming text only (successful):

1. → transcript with text to handle (required)
2. ← handled-start (required)
3. ← handled-chunk (required)
   • Chunk of response text
4. ← handled (required)
   • Sent for backwards compatibility
5. ← handled-stop (required)

Audio Output

1. → audio-start (required)
2. → audio-chunk (required)
   • One or more audio chunks
3. → audio-stop (required)
4. ← played