laser-asd 0.1.0

LASER ASD - Lip Landmark Assisted Speaker Detection for Active Speaker Detection

LASER ASD - Lip Landmark Assisted Speaker Detection

A PyTorch implementation of LASER ASD for Active Speaker Detection, providing a simple interface for the LoCoNet encoder with LASER landmark injection.

Features

• Active Speaker Detection using audio-visual fusion
• Based on LoCoNet architecture with LASER landmark injection
• Simple Python API for inference
• GPU acceleration with CUDA support
• Compatible with PyTorch 1.10+

Installation

pip install laser-asd

Quick Start

import numpy as np
from laser_asd import LaserASDModel

# Initialize model
model = LaserASDModel(device="cuda")

# Load pre-trained weights
model.load_weights("/path/to/loconet_laser.model")

# Prepare inputs
# face_crops: numpy array of shape [T, H, W, C] or [T, H, W]
# audio_data: numpy array of shape [samples] at 16kHz
face_crops = np.random.rand(100, 112, 112, 3).astype(np.float32)
audio_data = np.random.rand(64000).astype(np.float32)  # 4 seconds at 16kHz

# Predict speaking scores
scores = model.predict(face_crops, audio_data, sample_rate=16000, fps=25.0)

# scores >= 0 indicates speaking
is_speaking = scores >= 0

Model Weights

Pre-trained model weights can be downloaded from:

• Original LASER ASD repository

Set the model path via environment variable:

export LASER_ASD_MODEL_PATH=/path/to/loconet_laser.model

Or pass it directly:

model = LaserASDModel(device="cuda", model_path="/path/to/loconet_laser.model")

API Reference

LaserASDModel

class LaserASDModel:
    def __init__(
        self,
        device: str = "cuda",
        model_path: Optional[Path] = None,
        use_landmarks: bool = False,
    ):
        """
        Initialize LASER ASD model.

        Args:
            device: Device to run model on ('cuda' or 'cpu')
            model_path: Path to model weights file
            use_landmarks: Whether to use landmark features (False uses zeros)
        """

    def load_weights(self, model_path: Optional[str] = None):
        """Load model weights."""

    def predict(
        self,
        face_crops: np.ndarray,
        audio_data: np.ndarray,
        sample_rate: int = 16000,
        fps: float = 25.0,
    ) -> np.ndarray:
        """
        Predict speaking probability for each frame.

        Args:
            face_crops: Face crop images [T, H, W, C] or [T, H, W]
            audio_data: Audio waveform [samples]
            sample_rate: Audio sample rate (default: 16000)
            fps: Video frame rate (default: 25.0)

        Returns:
            Per-frame speaking scores [T] (>= 0 means speaking)
        """

Factory Function

def create_laser_model(
    device: str = "cuda",
    model_path: Optional[str] = None,
    **kwargs,
) -> LaserASDModel:
    """Factory function to create LASER ASD model."""

Requirements

• Python >= 3.8
• PyTorch >= 1.10.0
• torchvision >= 0.11.0
• numpy >= 1.20.0
• opencv-python >= 4.5.0
• resampy >= 0.4.0
• python_speech_features >= 0.6

Citation

If you use this code, please cite the original LASER ASD paper:

@inproceedings{nguyen2024laser,
  title={LASER: Lip Landmark Assisted Speaker Detection},
  author={Nguyen, Phat Lam and others},
  booktitle={Proceedings},
  year={2024}
}

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• Original LASER ASD implementation: https://github.com/plnguyen2908/LASER_ASD
• LoCoNet encoder architecture
• TalkNet audio-visual framework

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.