furhat-realtime-api 0.1.3

A Python client for the Furhat Realtime API.

Python Furhat Realtime API Client

Python implementation of Furhat Realtime API client.

There are two different clients:

• FurhatClient: A synchronous (blocking) client for simple, sequential command execution.
• AsyncFurhatClient: An asynchronous client for advanced, concurrent, or event-driven interactions.

Installation

Install the module with pip:

pip install furhat-realtime-api

Then include it your code:

from furhat_realtime_api import *

Synchronous Event client

The synchronous client (FurhatClient) is a wrapper around the asynchronous client that allows for blocking calls in a standard (non-async) Python context. This is useful for simple use cases where you want to send one command at a time and wait for the result before proceeding. Note that it is not suitable for advanced scenarios requiring concurrent or overlapping commands.

from furhat_realtime_api import FurhatClient
import logging

furhat = FurhatClient("127.0.0.1")  # Add authentication key as second argument if needed
furhat.set_logging_level(logging.INFO)  # Use logging.DEBUG for more details
furhat.connect()
furhat.request_speak_text("Hello world, I am Furhat.")
furhat.disconnect()

Asyncronous Event client

The asynchronous client is based on the Python asyncio library. This allows for much more flexible use cases, where requests can be aborted and responses can be streamed while actions are being executed.

from furhat_realtime_api import AsyncFurhatClient
import asyncio
import logging

furhat = AsyncFurhatClient("127.0.0.1")

async def run_example():
    await furhat.connect()
    await furhat.request_speak_text("Hello world, I am Furhat.", wait=True)
    await furhat.disconnect()

asyncio.run(run_example())

Note that one important difference when making requests in the asynchronous API is that they need to be called with the await keyword.

If you want to mix the use of the synchronous and asynchronous clients, you should not instantiate both, but you can access it like this:

from furhat_realtime_api import FurhatClient
import logging

furhat = FurhatClient("127.0.0.1")  
furhat_async = furhat.async_client
furhat.connect()

async def run_example():
    await furhat_async.request_speak_text("Hello world, I am Furhat.", wait=True)

asyncio.run(run_example())

Reacting to events

With the asynchronous client, you can react to events from the robot. This allows your application to react to specific events from the the robot, such as when the robot starts or stops speaking, hears something, or detects a user.

Here is an example of how to react to events related to the robot speaking:

from furhat_realtime_api import AsyncFurhatClient, Events
import asyncio

furhat = AsyncFurhatClient("127.0.0.1")

async def on_speak_start(event):
    print("Furhat started speaking:", event)

async def on_speak_end(event):
    print("Furhat finished speaking:", event)

async def main():
    await furhat.connect()
    # Register handlers
    furhat.add_handler(Events.response_speak_start, on_speak_start)
    furhat.add_handler(Events.response_speak_end, on_speak_end)
    
    await furhat.request_speak_text("Hello! Please say something.")
    await furhat.disconnect()

asyncio.run(main())

API reference

Here is a summary of the main methods. All of them are available in the asynchronous API, but only some of them in the synchronous wrapper API.

connect() / disconnect()

• Establishes or closes the connection to the robot.

Speaking (text-to-speech)

request_speak_text(text: str, wait: bool, abort: bool)

• Sync and Async: Makes the robot speak the given text. If wait is True, the method blocks until speaking is finished. If abort is True, any ongoing speech is stopped before starting the new one.

request_speak_audio(url: str, wait: bool, abort: bool, text: str, lipsync: bool = True)

• Sync and Async: Makes the robot play the given audio (from url). Same wait and abort logic as speak_text. text is just for logging purposes and can be omitted. lipsync denotes whether automatic lipsync should be added.

request_speak_stop()

• Sync and Async: Immediately stops any ongoing speech or audio playback.

Event callbacks (Async only)

response_speak_start

• The robot started to speak
• Parameters:
  • text (string): The synthesized text
  • gen_time (number): The time it took to synthesize (in ms)

response_speak_end

• The robot stopped speaking
• Parameters:
  • text (string): The synthesized text (what was actually said if aborted)
  • aborted (boolean, optional): Whether the speech was stopped prematurely (default: false)
  • failed (boolean, optional): Whether the speech synthesis failed (default: false)

response_speak_word

• Sent while speaking, if monitor_words is enabled in the speak request
• Parameters:
  • word (string): The word spoken
  • index (number): The index of the word spoken

response_speak_audio_buffer

• Audio buffer status update during streaming audio playback
• Parameters:
  • played (int): The number of bytes of audio data played
  • received (int): The number of bytes of audio data received

Speaking (streaming audio to the robot)

You can send audio to the robot which will be played through the robot's speakers. You can choose whether to add automatic lipsync or not.

request_speak_audio_start(sample_rate: int = 24000, lipsync: bool = False)

• Async only: Start streaming audio data to the robot for real-time speech synthesis.

request_speak_audio_data(audio: str)

• Async only: Send audio data chunk to the robot (used with request_speak_audio_start).

request_speak_audio_end()

• Async only: End the audio streaming session started with request_speak_audio_start.

Listening (speech-to-text)

request_listen_config(languages: list = ["en-US"], phrases: list = None)

• Sync and Async: Configure speech recognition languages and optional phrase hints for better recognition.

request_listen_start(partial: bool = False, concat: bool = True, stop_no_speech: bool = True, stop_robot_start: bool = True, stop_user_end: bool = True, resume_robot_end: bool = False, no_speech_timeout: float = 8.0, end_speech_timeout: float = 1.0)

• Starts listening for user speech with various configuration options for when to start/stop and timeout behavior.
• Sync: Will return the result as a string
• Async: You need to add event callbacks to get results

request_listen_stop()

• Async only: Force the robot to stop listening for speech.

Event callbacks (Async only)

You can add the following events handlers to receive results and monitor the speaking activity:

response_listen_start

• The robot started to listen

response_listen_end

• The robot stopped listening
• Parameters:
  • cause (string): The reason for stopping, can be either 'stopped', 'robot_speak', 'speech_end', 'silence_timeout'

response_hear_start

• The robot started to hear speech

response_hear_end

• The robot stopped hearing speech, with a speech recognition result
• Parameters:
  • text (string): Recognized speech

response_hear_partial

• The robot detected a partial speech recognition result
• Parameters:
  • text (string): Recognized speech

Voice

request_voice_status()

• Sync and Async: Returns the current voice configuration and status of the robot.

request_voice_config(voice_id: str, language: str, provider: str, gender: str, name: str, input_language: bool = True)

• Sync and Async: Configure the robot's voice according to the parameters (one or several can be used to add constraints).

Attention

request_attend_user(user_id: str)

• Sync and Async: Makes the robot turn its attention to a specific user (by user ID). Use the string closest to attend to the closest user.

request_attend_location(x: float, y: float, z: float)

• Sync and Async: Makes the robot attend to a specific location.

Gestures

request_gesture_start(name: str, intensity: float = 1.0, duration: float = 1.0, wait: bool = False)

• Sync and Async: Makes the robot perform a gesture. You can control the intensity and duration. If wait is True, the method blocks until the gesture is finished.

Event callbacks (Async only)

response_gesture_start

• The robot started performing a gesture

response_gesture_end

• The robot finished performing a gesture

Face

request_face_params(params: dict)

• Sync and Async: Sets facial animation parameters directly. The params dict should specify the desired facial features and their values.

request_face_headpose(yaw: float, pitch: float, roll: float, relative: bool)

• Sync and Async: Directly controls the head pose of the robot. Set relative to True for movement relative to current gaze (attention) target, or False for absolute positioning.

request_face_config(face_id: str = None, visibility: bool = None, microexpressions: bool = None)

• Sync and Async: Sets the current mask and character (face), and/or face visibility and microexpressions.

request_face_status(face_id: bool = True, face_list: bool = True)

• Sync and Async: Gets the current and available masks and characters. Returns information about the face configuration.

request_face_reset()

• Sync and Async: Resets all facial parameters to their default values.

LED

request_led_set(color: str)

• Sync and Async: Sets the color of the robot's LED. The color can be a color name (e.g., "red") or a hex code (e.g., "#FF0000").

Users

request_users_once()

• Sync and Async: Get the current user status and detected users as a one-time request. Returns the user object.

request_users_start()

• Async only: Start continuous monitoring of users in the robot's field of view. You must register an event callback to receive the user objects.

request_users_stop()

• Async only: Stop the continuous user monitoring started with request_users_start.

Event callbacks (Async only)

response_users_data

• Provides continuous updates about detected users in the robot's field of view, including user positions, IDs, and status information
• Parameters:
  • users (list): List of user objects, sorted by how close they are to the robot

Audio Streaming

request_audio_start(sample_rate: int = 16000, microphone: bool = True, speaker: bool = False)

• Async only: Start streaming audio data from the robot's microphone and/or to the robot's speaker. You must register an event callback to receive the audio.

request_audio_stop()

• Async only: Stop the audio streaming session started with request_audio_start.

Event callbacks (Async only)

response_audio_data

• Provides streaming audio data from the robot's microphone when audio streaming is active
• Parameters:
  • microphone (string, optional): Base64-encoded audio data, 16-bit, mono, little-endian
  • speaker (string, optional): Base64-encoded audio data, 16-bit, mono, little-endian

Camera

request_camera_once()

• Sync and Async: Get a one-time snapshot from the robot's camera. Returns the image data.

request_camera_start()

• Async only: Start streaming camera frames from the robot's camera. You must register an event callback to receive the images.

request_camera_stop()

• Async only: Stop the camera streaming started with request_camera_start.

Event callbacks (Async only)

response_camera_data

• Provides streaming camera frames from the robot's camera when camera streaming is active
• Parameters:
  • image (string): Base64-encoded jpeg image

Utility Methods

set_logging_level(level: int)

• Sync and Async: Set the logging level for the client (e.g., logging.DEBUG, logging.INFO).

add_handler(event_type: str | list[str], handler: Callable)

• Async only: Register an event handler for one or more event types.

remove_handler(event_type: str | list[str], handler: Callable)

• Async only: Remove a previously registered event handler.

wait_for_event(event_type: str | list[str], timeout: float = 5.0, request_id: str = None)

• Async only: Wait for a specific event type and return its data.

send_event(event: Dict)

• Async only: Send a raw event dictionary to the websocket.

send_event_and_wait(event: Dict, return_type: str | list[str], timeout: float = 5.0)

• Async only: Send an event and wait for a specific response type.