Collection of Preprocessing Functions and Deep Learning Methods for Multimodal Feature Extraction
Exordium is a comprehensive toolkit for multimodal feature extraction across audio, video, and text modalities. It provides preprocessing functions, utility tools, and deep learning wrappers for processing and analyzing multimodal data.
Features
Audio
| Functionality | Model / Method | Output |
|---|---|---|
| I/O | load, save, resample | waveform |
| Spectral features | MFCC, Mel-spectrogram (with pre-emphasis) | spectrogram |
| Low-level descriptors | OpenSMILE — eGeMAPSv02 | 88-d vector |
| Audio–language embeddings | CLAP (laion/larger_clap_music_and_speech) | 512-d vector |
| Speech representations | Wav2Vec2 (base-960h / emotion-iemocap) | (T, 768) |
| Speech representations | WavLM (microsoft/wavlm-base/base+/large) | (T, 768/1024) per layer |
| Speech emotion features | emotion2vec+ (emotion2vec_plus_seed) | (T, 768) |
Video
Face Detection & Tracking
| Functionality | Model / Method | Output |
|---|---|---|
| Face detection | YOLOv8-Face (arnabdhar/YOLOv8-Face-Detection) | bounding boxes |
| Face detection + keypoints | YOLO11-pose (yolo11n/s-pose_widerface) | bounding boxes + 5-pt keypoints |
| Multi-face tracking | IoU-based tracker | track IDs across frames |
| Face-ID tracking | AdaFace (IResNet-18/50/101, CVPR 2022) + IoU gating | track IDs with identity recovery |
| Protagonist extraction | IoU tracklets + AdaFace global clustering (agglomerative, FaceClusterTracker) |
single protagonist track |
| Streaming tracking | StreamingFacePipeline: detector + online tracker, frame by frame |
track IDs, provisional protagonist, stability flag |
Face Analysis
| Functionality | Model / Method | Output |
|---|---|---|
| Dense facial landmarks | MediaPipe FaceMesh (face_landmarker.task) | 478 × (x, y) |
| Iris landmarks | MediaPipe Iris | 71 eye pts + 5 iris pts, EAR, diameters |
| Head pose | 6DRepNet (300W-LP + AFLW2000) | yaw, pitch, roll (degrees) |
| Gaze estimation | L2CS-Net (ResNet-50, MPIIFaceGaze) | pitch, yaw (radians) |
| Gaze target estimation | Gaze-LLE (DINOv2 ViT-B/L, GazeFollow / ChildPlay) | 64×64 target heatmap + in/out-of-frame score per person |
| Facial action units | OpenGraphAU (Swin-T backbone) | 41-dim AU intensity vector |
Deep Visual Features
| Functionality | Model / Method | Output |
|---|---|---|
| Video features | Swin Transformer (tiny/small/base) | 768-d / 768-d / 1024-d |
| Face identity embeddings | AdaFace (IResNet-18/50/101, CVPR 2022) | 512-d L2-normalised |
| Face appearance features | FAb-Net | 256-d |
| Vision–language embeddings | CLIP (ViT-H/14, laion2B) | 1024-d |
| Self-supervised visual features | DINOv2 (small/base/large/giant) | 384 / 768 / 1024 / 1536-d |
| Facial expression features | EmotiEffNet (EfficientNet-B0/B2, AffectNet) | 1280-d / 1408-d |
| Universal face features | FaRL (CLIP ViT-B/16, LAION-Face 20M, CVPR 2022) | 512-d |
| Facial video features | MARLIN (ViT, 16-frame clips, CVPR 2023) | 384 / 768 / 1024-d |
Text
| Functionality | Model / Method | Output |
|---|---|---|
| Speech-to-text | Whisper (OpenAI) | transcript |
| Word-level timestamps | Whisper + torchaudio MMS_FA or whisperX forced alignment | word (start, end, score) |
| Transcript re-alignment | fuzzy search of known transcript in word stream (rapidfuzz) | segment (start, end, score) |
| Annotation validation | SpeechAlignmentPipeline facade — transcribe, align, evaluate, re-segment |
metrics + accept/recut/drop decisions |
| Transcript metrics | TranscriptEvaluator — raw + Whisper-normalized WER/CER, rapidfuzz similarity, xml-roberta semantic cosine |
TranscriptMetrics + is_acceptable |
| Contextual embeddings | BERT (bert-base-uncased) | (T, 768) |
| Contextual embeddings | RoBERTa (roberta-large) | (T, 1024) |
| Multilingual embeddings | XML-RoBERTa (xlm-roberta-base) | (T, 768) |
Utilities
- Device management — GPU/CPU selection via
get_torch_device - Caching —
@load_or_createdecorator (safetensors, npy, pkl, fdet, vdet, track) - Normalization — global, per-feature, sliding-window
- Padding — fixed-length sequence padding and masking
- Loss functions — Bell, ecl1 losses
- Concurrency — thread- and process-pool helpers
Installation
Requires uv.
uv pip install exordium # base only
uv pip install exordium[all] # all optional dependencies
uv pip install exordium[audio] # audio extras only
uv pip install exordium[video] # video extras only
uv pip install exordium[text] # text extras only (includes whisperX)
Install uv if you don't have it yet:
curl -LsSf https://astral.sh/uv/install.sh | sh
Extras
| Extra | Dependencies |
|---|---|
audio |
OpenSMILE, torchaudio — audio feature extraction |
text |
transformers, torchaudio, rapidfuzz, whisperX — text, speech & alignment |
video |
MediaPipe, Ultralytics, timm — face & video models |
all |
all previously described extras |
Development
git clone https://github.com/fodorad/exordium
cd exordium
uv pip install -e ".[all,dev]"
make check # lint + type-check + test + docs
Documentation
Related Projects
EmotionLinMulT (202X)
Efficient, transformer-based, multi-task emotion detection system.
- Paper: not published yet
- Code: github.com/fodorad/EmotionLinMulT
BlinkLinMulT (2023)
Transformer-based eye blink detection and eye state recognition across 7 public benchmark databases.
PersonalityLinMulT (2022)
LinMulT trained for Big Five personality trait estimation and sentiment analysis.
- Paper: Multimodal Sentiment and Personality Perception Under Speech
- Code: github.com/fodorad/PersonalityLinMulT
LinMulT
General-purpose multimodal transformer with linear-complexity attention mechanisms.
- Website: adamfodor.com/LinMulT
- Code: github.com/fodorad/LinMulT
Contact
Ádám Fodor — adamfodor.com · fodorad201@gmail.com
1 maintainer
Metadata
1 maintainer
Release files for exordium 2.14.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| exordium-2.14.0.tar.gz | 39.4 MB | With provenance | Details for exordium-2.14.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| exordium-2.14.0-py3-none-any.whl | Python 3 | none | any | Details for exordium-2.14.0-py3-none-any.whl |
Total release size: 39.6 MB
Release files / exordium-2.14.0.tar.gz
| Download URL | exordium-2.14.0.tar.gz |
|---|---|
| Size | 39.4 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
cfbfe58ae5c54bf9ad3bedbfe1eb78ccc4fb8d0f758f1c7b06898dcdd433bca0
|
|
BLAKE2b-256 checksum How to use checksums |
939cb1a4cd2becf2621e986fb64f7ae4eccd7dc8c9a5a61a1d26d6b02774972e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.
Transparency logRelease files / exordium-2.14.0-py3-none-any.whl
| Download URL | exordium-2.14.0-py3-none-any.whl |
|---|---|
| Size | 218.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ed9ae1efa8adbf24c42b94d54bd6027ca83ecee83fd8f796a1f85f9e0795803a
|
|
BLAKE2b-256 checksum How to use checksums |
e6b77a2cab11e02a096098c2f759e4039c5892b35b6edcf44bd01c3813118c2c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.
Transparency logSecurity for exordium 2.14.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 2 files in this release have been uploaded with provenance metadata.
Attestation for exordium-2.14.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
