multimodal-emotion-engine 0.2.0

Multimodal emotion recognition engine (audio + text + fusion)

Multimodal Emotion Recognition Engine combining audio, speech-to-text (ASR), and text-based emotion classification, with a simple CLI and Python API.

This project is designed as a modular engine that can: - 🎙️ Listen from microphone - 📂 Analyze audio files - 🧠 Fuse audio + text emotion signals - ⚡ Run locally with pretrained models

—

Features

• Audio emotion recognition - MFCC + pitch based model

• Speech-to-text (ASR) - Powered by faster-whisper

• Text emotion classification - DistilRoBERTa emotion model

• CLI interface - emotion-engine listen - emotion-engine analyze <file>

• Python API

• PyScaffold-based project structure

• Tested with pytest

—

Project Structure

multimodal_emotion_engine/
├── src/multimodal_emotion_engine/
│   ├── audio/
│   │   ├── emotion_model.py
│   │   ├── features.py
│   │   └── mic.py
│   ├── text/
│   │   └── asr.py
│   ├── engine.py
│   ├── cli.py
│   └── config.py
├── models/
│   └── emotion_model.pt
├── tests/
├── pyproject.toml
└── setup.cfg

—

Installation

### 1. Clone the repository

git clone https://github.com/satsin06/multimodal_emotion_engine.git
cd multimodal_emotion_engine

—

### 2. Create and activate a virtual environment (recommended)

python -m venv venv
source venv/bin/activate   # macOS / Linux
venv\Scripts\activate      # Windows

—

### 3. Install the package

#### Minimal install

pip install -e .

#### With audio support

pip install -e ".[audio]"

#### With ASR support

pip install -e ".[asr]"

#### Full installation (recommended)

pip install -e ".[full]"

—

CLI Usage

After installation, the CLI command is available as:

emotion-engine --help

### 🎙️ Listen from microphone

emotion-engine listen

This will: - Record audio from your microphone - Run emotion recognition - Print emotion probabilities

Press Ctrl+C to stop recording (microphone is safely closed).

—

### 📂 Analyze an audio file

emotion-engine analyze path/to/audio.wav

—

Python API Usage

You can also use the engine programmatically:

from multimodal_emotion_engine.engine import EmotionEngine

engine = EmotionEngine()
result = engine.analyze_file("sample.wav")

print(result)

Example output:

{
    "neutral": 0.38,
    "happy": 0.10,
    "sad": 0.11,
    "angry": 0.05
}

—

Models

• Audio emotion model - Stored in: models/emotion_model.pt

• Text emotion model - j-hartmann/emotion-english-distilroberta-base (downloaded automatically)

• ASR model - Faster-Whisper (downloaded on first use)

Models are cached under:

~/.cache/emotion-engine/

—

Running Tests

Install test dependencies:

pip install -e ".[dev]"

Run tests:

pytest

—

Development Notes

• Python ≥ 3.9

• PyScaffold 4.6

• Type hints enabled (Pylance / MyPy friendly)

• Modular design for future multimodal fusion

—

Roadmap

• [ ] Real-time streaming emotion detection

• [ ] Multilingual ASR

• [ ] Emotion fusion weighting

• [ ] REST API

• [ ] Model retraining pipeline

—

License

MIT License. See LICENSE.txt for details.

—

Author

Satyam Sinha

• GitHub: https://github.com/satsin06

• Email: [e-mail]