LASER ASD - Lip Landmark Assisted Speaker Detection (PyPI Package)

Note: This is a PyPI packaging wrapper for easy installation.

Original Repository: https://github.com/plnguyen2908/LASER_ASD

All credit for the LASER ASD model architecture and training goes to the original authors.

This package provides a simple Python API for the LASER ASD (Lip Landmark Assisted Speaker Detection) model, enabling easy installation via pip and straightforward inference.

Original Work

This package is based on the LASER ASD implementation by Phat Lam Nguyen et al.:

• Original Repository: https://github.com/plnguyen2908/LASER_ASD
• Paper: LASER: Lip Landmark Assisted Speaker Detection

If you use this code, please cite the original work (see Citation section).

Features

• Active Speaker Detection using audio-visual fusion
• Based on LoCoNet architecture with LASER landmark injection
• Simple Python API for inference
• GPU acceleration with CUDA support
• Compatible with PyTorch 1.10+

Installation

pip install laser-asd

Quick Start

import numpy as np
from laser_asd import LaserASDModel

# Initialize model
model = LaserASDModel(device="cuda")

# Load pre-trained weights (download from original repo)
model.load_weights("/path/to/loconet_laser.model")

# Prepare inputs
# face_crops: numpy array of shape [T, H, W, C] or [T, H, W]
# audio_data: numpy array of shape [samples] at 16kHz
face_crops = np.random.rand(100, 112, 112, 3).astype(np.float32)
audio_data = np.random.rand(64000).astype(np.float32)  # 4 seconds at 16kHz

# Predict speaking scores
scores = model.predict(face_crops, audio_data, sample_rate=16000, fps=25.0)

# scores >= 0 indicates speaking
is_speaking = scores >= 0

Model Weights

Pre-trained model weights must be downloaded from the original repository:

• Download: https://github.com/plnguyen2908/LASER_ASD

Set the model path via environment variable:

export LASER_ASD_MODEL_PATH=/path/to/loconet_laser.model

Or pass it directly:

model = LaserASDModel(device="cuda", model_path="/path/to/loconet_laser.model")

API Reference

LaserASDModel

class LaserASDModel:
    def __init__(
        self,
        device: str = "cuda",
        model_path: Optional[Path] = None,
        use_landmarks: bool = False,
    ):
        """
        Initialize LASER ASD model.

        Args:
            device: Device to run model on ('cuda' or 'cpu')
            model_path: Path to model weights file
            use_landmarks: Whether to use landmark features (False uses zeros)
        """

    def load_weights(self, model_path: Optional[str] = None):
        """Load model weights."""

    def predict(
        self,
        face_crops: np.ndarray,
        audio_data: np.ndarray,
        sample_rate: int = 16000,
        fps: float = 25.0,
    ) -> np.ndarray:
        """
        Predict speaking probability for each frame.

        Args:
            face_crops: Face crop images [T, H, W, C] or [T, H, W]
            audio_data: Audio waveform [samples]
            sample_rate: Audio sample rate (default: 16000)
            fps: Video frame rate (default: 25.0)

        Returns:
            Per-frame speaking scores [T] (>= 0 means speaking)
        """

Factory Function

def create_laser_model(
    device: str = "cuda",
    model_path: Optional[str] = None,
    **kwargs,
) -> LaserASDModel:
    """Factory function to create LASER ASD model."""

Requirements

• Python >= 3.8
• PyTorch >= 1.10.0
• torchvision >= 0.11.0
• numpy >= 1.20.0
• opencv-python >= 4.5.0
• resampy >= 0.4.0
• python_speech_features >= 0.6

Citation

If you use this code, please cite the original LASER ASD paper:

@inproceedings{nguyen2024laser,
  title={LASER: Lip Landmark Assisted Speaker Detection},
  author={Nguyen, Phat Lam and others},
  booktitle={Proceedings},
  year={2024}
}

License

This project is licensed under the MIT License - see the LICENSE file for details.

The original LASER ASD code is also MIT licensed.

Acknowledgments

• Original LASER ASD: https://github.com/plnguyen2908/LASER_ASD (Phat Lam Nguyen et al.)
• LoCoNet encoder architecture
• TalkNet audio-visual framework

Differences from Original Repository

This PyPI package adds:

• wrapper.py: Simple inference API (LaserASDModel class)
• pyproject.toml: PyPI packaging configuration
• Modified imports for package compatibility

The core model architecture and weights are unchanged from the original.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

For issues related to the model architecture, please refer to the original repository.