exordium 2.4.1

Collection of utility tools and deep learning methods for multimodal feature extraction.

Collection of Preprocessing Functions and Deep Learning Methods for Multimodal Feature Extraction

---

Exordium is a comprehensive toolkit for multimodal feature extraction across audio, video, and text modalities. It provides preprocessing functions, utility tools, and deep learning wrappers for processing and analyzing multimodal data.

Features

Audio

Functionality	Model / Method	Output
I/O	load, save, resample	waveform
Spectral features	MFCC, Mel-spectrogram (with pre-emphasis)	spectrogram
Low-level descriptors	OpenSMILE — eGeMAPSv02	88-d vector
Audio–language embeddings	CLAP (laion/larger_clap_music_and_speech)	512-d vector
Speech representations	Wav2Vec2 (base-960h / emotion-iemocap)	(T, 768)
Speech representations	WavLM (microsoft/wavlm-base/base+/large)	(T, 768/1024) per layer
Speech emotion features	emotion2vec+ (emotion2vec_plus_seed)	(T, 768)

Video

Face Detection & Tracking

Functionality	Model / Method	Output
Face detection	YOLOv8-Face (arnabdhar/YOLOv8-Face-Detection)	bounding boxes
Face detection + keypoints	YOLO11-pose (yolo11n/s-pose_widerface)	bounding boxes + 5-pt keypoints
Multi-face tracking	IoU-based tracker	track IDs across frames
Face-ID tracking	AdaFace (IResNet-18/50/101, CVPR 2022) + IoU gating	track IDs with identity recovery

Face Analysis

Functionality	Model / Method	Output
Dense facial landmarks	MediaPipe FaceMesh (face_landmarker.task)	478 × (x, y)
Iris landmarks	MediaPipe Iris	71 eye pts + 5 iris pts, EAR, diameters
Head pose	6DRepNet (300W-LP + AFLW2000)	yaw, pitch, roll (degrees)
Gaze estimation	L2CS-Net (ResNet-50, MPIIFaceGaze)	pitch, yaw (radians)
Gaze estimation	UniGaze (ViT-based)	pitch, yaw (radians)
Eye blink detection	BlinkDenseNet121 (DenseNet-121)	per-eye open/closed probability
Facial action units	OpenGraphAU (Swin-T backbone)	41-dim AU intensity vector

Deep Visual Features

Functionality	Model / Method	Output
Video features	Swin Transformer (tiny/small/base)	768-d / 768-d / 1024-d
Face identity embeddings	AdaFace (IResNet-18/50/101, CVPR 2022)	512-d L2-normalised
Face appearance features	FAb-Net	256-d
Vision–language embeddings	CLIP (ViT-H/14, laion2B)	1024-d
Self-supervised visual features	DINOv2 (small/base/large/giant)	384 / 768 / 1024 / 1536-d
Facial expression features	EmotiEffNet (EfficientNet-B0/B2, AffectNet)	1280-d / 1408-d
Facial video features	MARLIN (ViT, 16-frame clips, CVPR 2023)	384 / 768 / 1024-d

Text

Functionality	Model / Method	Output
Speech-to-text	Whisper (OpenAI)	transcript
Contextual embeddings	BERT (bert-base-uncased)	(T, 768)
Contextual embeddings	RoBERTa (roberta-large)	(T, 1024)
Multilingual embeddings	XML-RoBERTa (xlm-roberta-base)	(T, 768)

Utilities

• Device management — GPU/CPU selection via get_torch_device
• Caching — @load_or_create decorator (safetensors, npy, pkl, fdet, vdet, track)
• Normalization — global, per-feature, sliding-window
• Padding — fixed-length sequence padding and masking
• Loss functions — Bell, ecl1 losses
• Concurrency — thread- and process-pool helpers

---

Installation

Requires uv. The video extras include unigaze, which pins timm==0.3.2 (broken with modern PyTorch). uv's override-dependencies in pyproject.toml silently upgrades it to timm>=1.0. Plain pip has no equivalent override mechanism and will fail to resolve this conflict.

uv pip install exordium          # base only
uv pip install exordium[all]     # all optional dependencies
uv pip install exordium[audio]   # audio extras only
uv pip install exordium[video]   # video extras only
uv pip install exordium[text]    # text extras only

Install uv if you don't have it yet:

curl -LsSf https://astral.sh/uv/install.sh | sh

Extras

Extra	Dependencies
audio	OpenSMILE, torchaudio — audio feature extraction
text	transformers, torchaudio — text and speech models
video	MediaPipe, Ultralytics, blinklinmult, unigaze, timm — face & video models
all	all previously described extras

---

Development

git clone https://github.com/fodorad/exordium
cd exordium
uv pip install -e ".[all,dev]"
make check   # lint + type-check + test + docs

---

Documentation

• API Reference
• Demo Notebooks

---

Related Projects

EmotionLinMulT (202X)

Efficient, transformer-based, multi-task emotion detection system.

• Paper: not published yet
• Code: github.com/fodorad/EmotionLinMulT

BlinkLinMulT (2023)

Transformer-based eye blink detection and eye state recognition across 7 public benchmark databases.

• Paper: BlinkLinMulT: Transformer-based Eye Blink Detection
• Code: github.com/fodorad/BlinkLinMulT

PersonalityLinMulT (2022)

LinMulT trained for Big Five personality trait estimation and sentiment analysis.

• Paper: Multimodal Sentiment and Personality Perception Under Speech
• Code: github.com/fodorad/PersonalityLinMulT

LinMulT

General-purpose multimodal transformer with linear-complexity attention mechanisms.

• Website: adamfodor.com/LinMulT
• Code: github.com/fodorad/LinMulT

---

Contact

Ádám Fodor — adamfodor.com · [e-mail]