respeak-ai 0.1.8

Modules required for conversation, released as a Python SDK.

respeak

Provides the modules required for conversation and is released as a Python SDK.

Install

pip install respeak-ai

# local development
pip install -e .

Optional extras:

pip install "respeak-ai[vad]"
pip install "respeak-ai[tts]"
pip install "respeak-ai[llm]"
pip install "respeak-ai[a2f]"
pip install "respeak-ai[a2f-trt]"   # Audio2Face + TensorRT (faster ONNX EP)
pip install "respeak-ai[dev]"

Supported models

Module	Class	Recommended weights	Source
ASR	StreamingParaformerAsr	paraformer-zh-streaming (iic/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online)	ModelScope · Hugging Face · FunASR
ASR (punc)	used via punc_model=	ct-punc (iic/punc_ct-transformer_cn-en-common-vocab471067-large)	ModelScope · Hugging Face
TTS	CosyVoice3Tts	Fun-CosyVoice3-0.5B-2512	ModelScope · Hugging Face · CosyVoice
VAD	SileroVad	Silero VAD (auto-download via silero-vad)	GitHub · PyPI
LLM	Qwen3LLM	Qwen3-8B (or other Qwen3 / Qwen2.5 Instruct checkpoints)	ModelScope · Hugging Face · Qwen
Audio2Face	NvidiaAudio2Face3D	Audio2Face-3D-v2.3.1-Claire	Hugging Face · Audio2Face-3D

Download examples:

# ASR + punctuation (ModelScope)
modelscope download --model iic/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online \
  --local_dir ./models/paraformer-zh-streaming
modelscope download --model iic/punc_ct-transformer_cn-en-common-vocab471067-large \
  --local_dir ./models/punc_ct-transformer_cn-en-common-vocab471067-large

# TTS (ModelScope or Hugging Face)
modelscope download --model FunAudioLLM/Fun-CosyVoice3-0.5B-2512 \
  --local_dir ./models/Fun-CosyVoice3-0.5B-2512

# LLM
modelscope download --model Qwen/Qwen3-8B --local_dir ./models/Qwen3-8B

# Audio2Face 3D (Hugging Face)
huggingface-cli download nvidia/Audio2Face-3D-v2.3.1-Claire \
  --local-dir ./models/audio2face-3d-v2.3.1-claire

Silero VAD needs no manual download: pip install "respeak-ai[vad]" then SileroVad.from_pretrained().

Usage

from respeak import StreamingParaformerAsr, CosyVoice3Tts, SileroVad, Qwen3LLM, NvidiaAudio2Face3D

# ASR
asr = StreamingParaformerAsr.from_pretrained(asr="path/or/model_id", punc_model="path/or/model_id")
text = asr.generate(input=audio_chunk, is_final=False, cache={})

# CosyVoice3 TTS (vLLM)
tts = CosyVoice3Tts.from_pretrained(
    model_dir="path/to/TTS_Model",
    prompt_text="...",
    prompt_wav="path/to/prompt.wav",
    load_vllm=True,
)
for wav in tts.generate("你好，欢迎使用本系统。", stream=True):
    ...

# VAD
vad = SileroVad.from_pretrained()
speech_prob = vad.generate(audio_chunk)

# LLM sentence-level streaming
llm = Qwen3LLM.from_pretrained(
    "path/to/LLM_Model_ft",
    base_prompt_file="path/to/base_prompt.md",
    persona_setting="你叫小睿，是一个温和的中文助手。",
    strategy_prompt="回答要简洁自然。",
)
for sentence in llm.generate("你好，介绍一下你自己。", stream=True):
    ...

# Audio2Face 3D (requires .[a2f])
a2f = NvidiaAudio2Face3D.from_pretrained("path/to/audio2face-3d-model", use_cuda=True)
frame = a2f.generate(audio_window)  # {"audio": [...], "arkit_weights": [51]}
frames = a2f.generate(audio, stream=True, is_final=True)  # list[{"audio", "arkit_weights"}]

# UE5 Live Link TCP rendering
from respeak.models.nvidia_audio2face_3d import Ue5BlendshapeRenderer
with Ue5BlendshapeRenderer(tcp_ip="127.0.0.1", tcp_port=12030) as renderer:
    renderer.stream_frames(frames, realtime=True)

New model types should live under src/respeak/models/<name>/ (subclass BaseModel, expose from_pretrained / generate). Add matching tests under tests/models/<name>/.

Examples

# ASR
python examples/paraformer_asr_streaming.py \
  --asr path/to/ASR_Model \
  --punc path/to/PUNC_Model \
  --audio path/to/test.wav

# TTS (requires .[tts])
python examples/cosyvoice3_tts_streaming.py \
  --model-dir path/to/Fun-CosyVoice3 \
  --prompt-wav path/to/prompt.wav \
  --prompt-text "..." \
  --text "你好，欢迎使用本系统。" \
  --output output.wav

# VAD (requires .[vad])
python examples/silerovad_real_inference.py \
  --audio path/to/test.wav \
  --threshold 0.5

# LLM (requires .[llm])
python examples/qwen3_llm_streaming.py \
  --model-path path/to/LLM_Model_ft \
  --base-prompt path/to/base_prompt.md \
  --text "你好，介绍一下你自己。"

# Audio2Face 3D (requires .[a2f]; add --tensorrt with .[a2f-trt])
python examples/nvidia_audio2face_3d_streaming.py \
  --model-dir path/to/audio2face-3d-v2.3.1-claire \
  --audio path/to/test.wav \
  --output arkit_weights.npy \
  --output-audio synced_audio.wav

# Stream to UE5 Audio2Face plugin
python examples/nvidia_audio2face_3d_streaming.py \
  --model-dir path/to/audio2face-3d-v2.3.1-claire \
  --audio path/to/test.wav \
  --ue5-tcp-ip 127.0.0.1 \
  --ue5-tcp-port 12030 \
  --ue5-realtime

Audio2Face GPU backends

Mode	Install	Run
CUDA (default)	pip install "respeak-ai[a2f]"	omit --tensorrt → CUDAExecutionProvider
TensorRT	pip install "respeak-ai[a2f-trt]"	add --tensorrt → TensorrtExecutionProvider (first run builds TRT engine cache under model dir)

ORT 1.23 pairs with TensorRT 10.9 (installed via the a2f-trt extra). Without the tensorrt pip package, --tensorrt falls back to CUDA automatically.

Tests

pip install -e ".[dev]"
pytest tests/models -q