attenlabs-sas

Python SDK for Attention Labs real-time selective auditory attention.

Every voice pipeline has the same problem: the microphone hears everything, but your ASR should only process speech directed at the device. Wake words solve this with a rigid trigger phrase. SAS solves it without one — classifying every audio frame as silent, human-directed, or device-directed and routing only what matters.

attenlabs-sas streams mic and webcam data to the SAS inference server over WebSocket and emits typed events: attention predictions, voice activity, conversation state, and ready-to-forward speech audio. LLM routing is left to you.

Sign up

Get your API token at attentionlabs.ai/dashboard.

Install

pip install attenlabs-sas

Requires Python 3.10+. sounddevice and opencv-python are pulled in automatically for mic and camera access.

Quickstart

import time
from sas import AttentionClient

client = AttentionClient(token="your-token")

@client.on_prediction
def _(event):
    label = {0: "silent", 1: "human", 2: "device"}.get(event.cls, "?")
    print(f"{label}  {event.confidence:.0%}  faces={event.num_faces}  src={event.source}")

@client.on_speech_ready
def _(event):
    # event.audio_base64 — base64 PCM16 @ 16 kHz mono, ready for OpenAI Realtime / any LLM
    # event.audio_pcm16  — same audio as np.int16 array
    print(f"speech ready ({event.duration_sec:.2f}s)")

@client.on_error
def _(event):
    print(f"ERROR: {event.title}: {event.message}")

client.start()
try:
    while True:
        time.sleep(0.1)
except KeyboardInterrupt:
    client.stop()

A full CLI demo wiring SAS + OpenAI Realtime lives at sas-py-demo.

API

AttentionClient

from sas import AttentionClient, CameraConfig, MicConfig

client = AttentionClient(
    token="...",                    # Auth token — sent as WS subprotocol
    url=None,                      # Server URL (default: wss://server.attentionlabs.ai/ws)
    video=CameraConfig(),          # Webcam config
    audio=MicConfig(),             # Mic config
    initial_threshold=0.7,         # Device-class confidence threshold (0..1)
    enable_audio=True,             # Set False to skip mic capture
    enable_video=True,             # Set False to skip webcam capture
)

Configuration

MicConfig

CameraConfig

Methods

Events

Register handlers with decorators. All callbacks fire on internal threads — keep them fast or hand work off to your own thread.

@client.on_prediction
def handle(event):
    ...

Event types

PredictionEvent

cls: int            # 0 = silent, 1 = human-directed, 2 = device-directed
confidence: float   # 0..1
source: str         # "video" or "audio"
num_faces: int      # faces detected in frame

VadEvent

probability: float  # VAD probability 0..1
is_speech: bool     # whether speech was detected

StateEvent

state: ConversationState  # "listening" | "sending" | "cancelled" | "idle"

SpeechReadyEvent

audio_pcm16: np.ndarray   # int16 array @ 16 kHz mono
audio_base64: str          # same audio as base64 — ready for OpenAI Realtime, etc.
duration_sec: float        # duration in seconds

ConfigEvent

model_class2_threshold: float  # server-confirmed threshold

StatsEvent

rtt_ms: float | None  # round-trip latency in ms
sent_video: int        # total video frames sent
skipped_video: int     # total video frames skipped
sent_audio: int        # total audio chunks sent
uptime_s: float        # connection uptime in seconds

AttentionErrorEvent

title: str                  # error category ("Auth Failed", "Connection Stalled", etc.)
message: str                # human-readable message
detail: str | None = None   # technical detail
code: int | None = None     # WebSocket close code, if applicable

DisconnectedEvent

code: int        # WebSocket close code
reason: str      # close reason
was_clean: bool  # True if code == 1000

LLM integration

LLM routing is intentionally not part of the SDK. The speech_ready event hands you PCM16 audio — both as a NumPy array and as base64 — forward it wherever you like.

When your LLM starts generating, call mute() + mark_responding(True) to suppress predictions during playback. When it finishes, unmute() + mark_responding(False).

from sas import AttentionClient

client = AttentionClient(token="...")

@client.on_speech_ready
def _(event):
    # Forward to your LLM of choice
    your_llm.send(event.audio_base64)

def on_llm_speaking():
    client.mute()
    client.mark_responding(True)

def on_llm_done():
    client.unmute()
    client.mark_responding(False)

See sas-py-demo for a full working example with OpenAI Realtime.

Threading model

The SDK manages four threads internally:

All event callbacks fire on sas-ws or sas-heartbeat. Don't block them — offload heavy work to your own thread.

License

MIT