cued-speech 0.3.1

Cued Speech Processing Tools - Decode and Generate cued speech videos

Cued Speech Processing Tools

A comprehensive Python package for processing cued speech videos with both decoding and generation capabilities. This package provides functionality to decode cued speech videos into subtitled output and generate cued speech videos from text input.

Features

Decoder Features

• Real-time Video Processing: Process cued speech videos using MediaPipe for landmark extraction
• Neural Network Inference: Use trained CTC models for phoneme recognition
• French Language Correction: Apply KenLM language models and homophone correction
• Subtitle Generation: Generate subtitled videos with French sentences

Generator Features

• Text-to-Cued Speech: Generate cued speech videos from French text input
• Whisper Integration: Automatic speech recognition for accurate alignment
• MFA Alignment: Montreal Forced Alignment for precise phoneme timing
• Hand Gesture Overlay: Realistic hand shape and position rendering
• Automatic Synchronization: Perfect alignment between speech and visual cues

Data Management Features

• Automatic Data Download: Automatically download required model files and data
• GitHub Release Integration: Seamless download from GitHub releases
• Smart Caching: Avoid re-downloading existing files
• Easy Cleanup: Simple commands to manage downloaded data

General Features

• Command Line Interface: Easy-to-use CLI for both decoding and generation
• Organized Output Structure: Separate folders for decoder and generator outputs
• Extensible Architecture: Modular design for future enhancements
• PyPI Ready: Ready for publication and easy installation

Installation

Prerequisites

• Python 3.11.*
• Pixi (to install Montreal Forced Aligner)

Install with Pixi (Recommended)

Use Pixi to install MFA, then install cued_speech via pip inside the Pixi environment.

1) Install Pixi

• macOS/Linux:

curl -fsSL https://pixi.sh/install.sh | bash

• Windows (PowerShell):

irm https://pixi.sh/install.ps1 | iex

More options: https://pixi.sh/installation/

2) Create a clean Pixi environment and install MFA

mkdir cued-speech-env && cd cued-speech-env
pixi init
pixi add "python==3.11"
pixi add montreal-forced-aligner=3.3.4
pixi run mfa version

3) Install the cued_speech package (pip inside Pixi)

pixi run python -m pip install cued-speech

4) Prepare French MFA Models (Required for Generation)

The cued speech generator requires French MFA models (acoustic + dictionary). These are now bundled with the data downloaded by the package. Just download the data, then save the models with MFA:

# Download all required data (includes MFA French models under ./download/)
pixi shell
cued-speech download-data

# Save the French acoustic model to MFA's model store (zip file)
pixi run mfa models save acoustic download/french_mfa.zip --overwrite

# Save the French dictionary model to MFA's model store (.dict file)
pixi run mfa models save dictionary download/french_mfa.dict --overwrite

Note:

• You can run the above inside a Pixi shell (pixi shell) or prefix with pixi run as shown.
• After saving, MFA will manage models in its own cache (e.g., ~/.local/share/mfa/models/).

5) Verify installation and see available options

pixi shell
cued-speech

Data Setup

The package requires several model files and data for operation. These are automatically downloaded on first use, but you can also manage them manually.

Manual Data Management

You can manage data files manually using the provided commands:

# Download all required data files, verify that you are in the pixi environment
cued-speech download-data 

# List available data files
cued-speech list-data

# Clean up downloaded data files
cued-speech cleanup-data --confirm

Required Data Files

The following files are automatically downloaded to a download/ folder in your current working directory:

• cuedspeech-model.pt - Pre-trained neural network model
• phonelist.csv - Phoneme vocabulary
• lexicon.txt - French lexicon
• kenlm_fr.bin - French language model
• homophones_dico.jsonl - Homophone dictionary
• kenlm_ipa.binary - IPA language model
• ipa_to_french.csv - IPA to French mapping
• test_decode.mp4 - Sample video for testing
• test_generate.mp4 - Sample video for generation
• rotated_images/ - Directory containing hand shape images for generation
• french_mfa.dict - MFA dictionary
• french_mfa.zip - MFA acoustic model Note: Data files are stored in ./download/ relative to where you run the commands, making them easy to find and manage.

Usage

Command Line Interface

The package provides a comprehensive command-line interface for both decoding and generating cued speech videos:

Note: The models are designed for videos at 30 FPS. For best results, use input videos that are 30 FPS.

Decoding (Cued Speech → Text)

Decode a cued speech video into a subtitled video.

Options:

• --video_path PATH (default: download/test_decode.mp4): Input cued-speech video
• --right_speaker [True|False] (default: True): Whether the speaker uses the right hand
• --model_path PATH (default: download/cuedspeech-model.pt): Pretrained model file
• --output_path PATH (default: output/decoder/decoded_video.mp4): Output subtitled video
• --vocab_path PATH (default: download/phonelist.csv): Vocabulary file
• --lexicon_path PATH (default: download/lexicon.txt): Lexicon file
• --kenlm_fr PATH (default: download/kenlm_fr.bin): KenLM model file
• --homophones_path PATH (default: download/homophones_dico.jsonl): Homophones dictionary
• --kenlm_ipa PATH (default: download/kenlm_ipa.binary): IPA language model
• --auto_download [True|False] (default: True): Auto-download missing data files

# Basic usage (uses default paths, automatically downloads data if needed)
cued-speech decode

# With custom video path
cued-speech decode --video_path /path/to/your/video.mp4

# Disable automatic data download
cued-speech decode --auto_download False

# Advanced usage with custom settings
cued-speech decode \
    --video_path /path/to/your/video.mp4 \
    --output_path output/decoder/my_decoded_video.mp4 \
    --model_path /path/to/custom_model.pt \
    --vocab_path /path/to/custom_vocab.csv \
    --lexicon_path /path/to/custom_lexicon.txt \
    --kenlm_fr /path/to/custom_kenlm.bin \
    --homophones_path /path/to/custom_homophones.jsonl \
    --kenlm_ipa /path/to/custom_lm.binary \
    --right_speaker True

Generation (Video → Cued Speech)

Generate a cued speech video from a video file. Text is extracted with Whisper unless --skip-whisper is used and --text is provided.

Arguments:

• VIDEO_PATH (positional): Path to input video file

Options:

• --text TEXT (default: None): Provide text manually (otherwise Whisper extracts it)
• --output_path PATH (default: output/generator/generated_cued_speech.mp4): Output video path
• --audio_path PATH (default: None): Optional audio file (extracted from video if not provided)
• --language [french|...] (default: french): Processing language
• --skip-whisper (flag): Skip Whisper download/transcription (requires --text)
• --whisper_model (deprecated): Use Python API to pass a loaded model object instead
• --easing [linear|ease_in_out_cubic|ease_out_elastic|ease_in_out_back] (default: ease_in_out_cubic): Gesture easing
• --morphing/--no-morphing (default: --morphing): Hand shape morphing
• --transparency/--no-transparency (default: --transparency): Transparency effects during transitions
• --curving/--no-curving (default: --curving): Curved trajectories

# Basic usage (text extracted automatically from video)
cued-speech generate input_video.mp4

# With custom output path
cued-speech generate speaker_video.mp4 --output_path output/generator/my_generated_video.mp4

# With custom audio file
cued-speech generate speaker_video.mp4 --audio_path custom_audio.wav

# With different language
cued-speech generate speaker_video.mp4 --language english

# Prefer Python API to pass a loaded Whisper model object

# With manual text (optional)
cued-speech generate speaker_video.mp4 --text "Merci beaucoup pour votre attention"

# Skip Whisper if you have SSL issues
cued-speech generate speaker_video.mp4 --skip-whisper --text "Merci beaucoup pour votre attention"

Output Structure

The package organizes outputs in a structured way:

output/
├── decoder/           # Decoded videos with subtitles
│   └── decoded_video.mp4
└── generator/         # Generated cued speech videos
    ├── audio.wav           # Extracted/processed audio
    ├── audio.TextGrid      # MFA alignment results
    ├── rendered_video.mp4  # Video with hand cues (no audio)
    ├── final_rendered_video.mp4  # Final output with audio
    └── mfa_input/          # MFA temporary files

Python API

You can also use the package programmatically:

Decoder API

from cued_speech import decode_video

# Decode a cued speech video
decode_video(
    video_path="input.mp4",
    right_speaker=True,
    model_path="/path/to/model.pt",
    output_path="output/decoder/decoded.mp4",
    vocab_path="/path/to/vocab.csv",
    lexicon_path="/path/to/lexicon.txt",
    kenlm_model_path="/path/to/kenlm.bin",
    homophones_path="/path/to/homophones.jsonl",
    lm_path="/path/to/lm.binary"
)

Generator API

from cued_speech import generate_cue
import whisper

# Generate a cued speech video (text extracted automatically)
model = whisper.load_model("medium", download_root="download")
result_path = generate_cue(
    text=None,  # Will be extracted from video using Whisper
    video_path="download/test_generate.mp4",  # Required: input video path
    output_path="output/generator/",  # Required: output video path
    audio_path=None,  # Will extract from video
    config={
        # Configurable parameters (defaults shown), redefine only those you need to modify. 
        # Others will be infered from the default config file.
        "handshapes_dir": "download/handshapes/coordinates",
        "language": "french",
        "reference_face_size": 0.3,
        "hand_scale_factor": 0.75,
        "mfa_args": ["--beam", "200", "--retry_beam", "400", "--fine_tune"],
        "video_codec": "libx265",
        "audio_codec": "aac",
        "easing_function": "linear",
        "enable_morphing": False,
        "enable_transparency": False,
        "enable_curving": True,
        "skip_whisper": True, # Only if you provide text manually, 
        "model": model, # Provide a preloaded Whisper model (optional). If None, defaults to medium.

    }
)
print(f"Generated video saved to: {result_path}")

# Or with manual text (minimal config)
result_path = generate_cue(
    text="Bonjour tout le monde",
    video_path="speaker_video.mp4",  # Required argument
    output_path="output/generator/generated.mp4"  # Required argument
)

Architecture

Core Components

Decoder Components

1. MediaPipe Integration: Extracts hand and lip landmarks from video frames
2. Feature Extraction: Processes landmarks into hand shape, position, and lip features
3. Neural Network: Three-stream fusion encoder with CTC output
4. Language Model: KenLM-based beam search for French sentence correction
5. Video Processing: Generates subtitled output with synchronized audio

Generator Components

1. Whisper Integration: Automatic speech recognition for transcription
2. MFA Alignment: Montreal Forced Alignment for precise phoneme timing
3. Cue Mapping: Maps phonemes to hand shapes and positions using cued speech rules
4. Hand Rendering: Overlays realistic hand gestures onto video frames
5. Synchronization: Ensures perfect timing between speech and visual cues

Model Architecture

Decoder Architecture

The decoder uses a three-stream fusion encoder:

• Hand Shape Stream: Processes hand landmark positions and geometric features
• Hand Position Stream: Analyzes hand movement and positioning
• Lips Stream: Extracts lip movement and facial features

Real-Time Decoding with Overlap-Save Windowing

The decoder implements a real-time processing strategy using an overlap-save method that maintains full bidirectional context while enabling streaming inference:

Windowing Parameters:

• Window Size: 100 frames (processed at once)
• Commit Size: 50 frames (kept from each window after first two chunks)
• Context: 25 frames left + 25 frames right

Processing Strategy:

1. Valid Frame Filtering: Only frames with complete features (hand visible, face detected) are counted and processed
2. Chunk Processing:
   • Chunk 0: Process frames [0-99], commit [0-49] (0L/50R context)
   • Chunk 1: Process frames [25-124], commit [50-74] (25L/50R context)
   • Chunk 2+: Process frames [50, 100, 150...], commit center 50 frames with full 25L/25R context
3. Incremental Decoding: After each chunk, decode using the full accumulated logits history for progressive sentence refinement
4. Bidirectional Context: All committed frames (except first 50 and final chunk) have full bidirectional context for optimal accuracy

Generator Architecture

The generator follows a multi-stage pipeline:

• Audio Processing: Whisper-based transcription and feature extraction
• Phoneme Alignment: MFA-based precise timing alignment
• Cue Generation: Rule-based mapping from phonemes to hand configurations
• Video Rendering: Real-time hand overlay with facial landmark tracking

Processing Pipeline

Decoding Pipeline

1. Video Input: Load and process video frames
2. Landmark Extraction: Use MediaPipe to extract hand and face landmarks
3. Feature Computation: Calculate geometric and temporal features
4. Model Inference: Run CTC model to predict phonemes
5. Language Correction: Apply beam search with language models
6. Subtitle Generation: Create output video with French subtitles

Generation Pipeline

1. Text Input: Process French text for cued speech generation
2. Audio Extraction: Extract or use provided audio track
3. Speech Recognition: Use Whisper for accurate transcription
4. Phoneme Alignment: Apply MFA for precise timing
5. Cue Mapping: Map phonemes to hand shapes and positions
6. Video Rendering: Overlay hand cues with perfect synchronization

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• MediaPipe for landmark extraction
• PyTorch for deep learning framework
• KenLM for language modeling
• The cued speech research community

Support

For questions and support:

• Contact: [e-mail]