pec-dss 0.1.0

Paralinguistic Event Classification from Diarized Speaker Segments

PEC-DSS 🎵🔊

English | 한국어 | 中文 | 日本語

Paralinguistic Event Classification from Diarized Speaker Segments

PEC-DSS is an advanced audio analysis system that identifies paralinguistic vocal events (like laughter, sighs, etc.) and attributes them to specific speakers through sophisticated speaker diarization and neural audio processing.

✨ Features

• 🎙️ Advanced speaker identification using neural audio encoders
• 😀 Attribution of paralinguistic events to specific speakers
• 🔍 High-accuracy SNAC (Scalable Neural Audio Codec) model integration
• 🔊 Voice embedding and similarity-based speaker matching
• 📊 Comprehensive audio codebook analysis
• 🔄 Modular architecture for easy customization

🚀 Installation

From PyPI

pip install pec-dss

From Source with pip

git clone https://github.com/hwk06023/PEC-DSS.git
cd PEC-DSS
pip install -e .

From Source with requirements.txt

git clone https://github.com/hwk06023/PEC-DSS.git
cd PEC-DSS
pip install -r requirements.txt

For development:

pip install -r requirements-dev.txt

📖 Quick Start

Basic Usage

from snac_model import load_snac_model
from audio_encoder import get_codebook_vectors
from speaker_identification import assign_speakers_to_laughs
import librosa

# Load SNAC model
snac_model = load_snac_model(device="cpu")  # or "cuda" for GPU

# Prepare speaker reference samples
speaker_samples = {
    "speaker1": [audio1, audio2],  # Audio waveforms as numpy arrays
    "speaker2": [audio3, audio4]
}

# Process unidentified audio events
unidentified_events = [event1, event2]  # Audio waveforms as numpy arrays

# Identify speakers for each audio event
results = assign_speakers_to_laughs(speaker_samples, unidentified_events, snac_model)

# Print results
for speaker, events in results.items():
    print(f"Speaker {speaker} has {len(events)} attributed events")

🧩 System Architecture

PEC-DSS consists of the following components:

• snac_model.py: SNAC model initialization and management
• audio_encoder.py: Audio encoding and vectorization
• codebook_analysis.py: Statistical analysis of audio codebooks
• speaker_identification.py: Speaker identification algorithms
• main.py: Integration and execution framework

🔊 Audio Event Types

The system can identify various paralinguistic events including:

• Laughter
• Sighs
• Crying
• Coughing
• Other non-verbal vocal expressions

🚀 Future Developments

• 🧠 Integration with more audio encoder models
• 😢 Expanded paralinguistic event recognition
• 🎵 Emotional tone classification
• ⚡ Performance optimization for real-time processing

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

📄 License

This project is licensed under the GNU General Public License v3.0.

🙏 Acknowledgements

• SNAC - Scalable Neural Audio Codec
• HuggingFace Transformers - Machine learning tools
• Llama - Language models for text processing