exordium 1.1.1

Collection of utility tools and deep learning methods.

exordium

Collection of preprocessing functions and deep learning methods.

Supported features

Audio

• frequently used io for audio files
• openSMILE feature extraction
• spectrogram calculation
• Wav2Vec2 feature extraction

Video

• frequently used io for videos and frames
• bounding box manipulation methods
• face detection with RetinaFace
• face landmarks and head pose with 3DDFA_V2
• body pose estimation with max-human-pose-estimator
• categorical and dimensional emotion estimation with EmoNet
• iris and pupil landmark estimation with MediaPipe Iris
• fine eye landmark estimation with MediaPipe FaceMesh
• tracking using IoU and DeepFace
• FAb-Net feature extraction
• OpenFace feature extraction
• R2+1D feature extraction

Text

• BERT feature extraction
• RoBERTa feature extraction

Utils

• parallel processing
• io decorators
• loss functions
• normalization

Visualization

• graphs
• 3D headpose
• 2D landmarks
• dataframes to images

Setup

Install package with PyPi

pip install exordium

Install package for development

git clone https://github.com/fodorad/exordium
cd exordium
git submodule update --init --recursive
pip install .
pip install -U -r requirements.txt

Run unittests

python -m unittest discover -s test

Projects using exordium

(2023) BlinkLinMulT

LinMulT is trained for blink presence detection and eye state recognition tasks. Our results demonstrate comparable or superior performance compared to state-of-the-art models on 2 tasks, using 7 public benchmark databases.

• paper: BlinkLinMulT: Transformer-based Eye Blink Detection (accepted, available soon)
• code: https://github.com/fodorad/BlinkLinMulT

(2022) PersonalityLinMulT

LinMulT is trained for Big Five personality trait estimation using the First Impressions V2 dataset and sentiment estimation using the MOSI and MOSEI datasets.

• paper: Multimodal Sentiment and Personality Perception Under Speech: A Comparison of Transformer-based Architectures (pdf, website)
• code: https://github.com/fodorad/PersonalityLinMulT

What's next

• Add support for Action Unit detection (OpenGraphAU)
• Add support for Gaze estimation (L2CS-Net)
• Add support for Blink estimation (DenseNet121, LinT, BlinkLinMulT)
• Add support for Personality trait estimation (PersonalityLinMulT)

Contact

• Ádám Fodor ([e-mail])