linmult 2.1.2

General-purpose Multimodal Transformer with Linear-Complexity Attention

LinMulT is a modular Transformer library designed for multimodal sequence modelling. It handles variable-length inputs across any number of modalities, supports missing-modality scenarios, and offers six attention variants ranging from O(N²) softmax to O(N·s) gated linear attention; all behind a single config file.

Features

Installation

pip install linmult

For development:

git clone https://github.com/fodorad/linmult
cd linmult
uv sync --extra dev --extra docs
make check

Quick start

LinT — single-modality transformer

import torch
from linmult import LinT

x = torch.rand(8, 1500, 25)  # (batch, time, features)

model = LinT({
    'input_feature_dim': 25,
    'heads': [{'name': 'out', 'type': 'simple', 'output_dim': 5}],
    'time_dim_reducer': 'attentionpool',  # aggregate over time
})
result = model(x)
assert result['out'].shape == (8, 5)

LinMulT — multimodal transformer

import torch
from linmult import LinMulT

x1 = torch.rand(8, 1500, 25)  # (batch, time, features)
x2 = torch.rand(8,  450, 35)
x3 = torch.rand(8,  450, 256)

model = LinMulT({
    'input_feature_dim': [25, 35, 256],
    'heads': [{'name': 'sentiment', 'type': 'simple', 'output_dim': 3}],
    'time_dim_reducer': 'gap',
})
result = model([x1, x2, x3])
assert result['sentiment'].shape == (8, 3)

Switching attention type

model = LinT({
    'input_feature_dim': 64,
    'heads': [{'name': 'out', 'type': 'simple', 'output_dim': 10}],
    'attention_type': 'flash',        # linear, performer, flash, bigbird, softmax, mha
    'flash_query_key_dim': 32,        # flash (GAU) scoring dimension
})

Documentation

API reference

Config reference

Quick-start notebook

Attention benchmark

UR-Funny training example

Similar projects using LinMulT

BlinkLinMulT (2023)

LinMulT trained for blink presence detection and eye state recognition across 7 public benchmark databases.

• Paper: BlinkLinMulT: Transformer-based Eye Blink Detection
• Code: github.com/fodorad/BlinkLinMulT

PersonalityLinMulT (2022)

LinMulT trained for Big Five personality trait estimation and sentiment analysis (MOSI, MOSEI, First Impressions V2).

• Paper: Multimodal Sentiment and Personality Perception Under Speech
• Code: github.com/fodorad/PersonalityLinMulT

Citation

If you found this work helpful, please cite the relevant paper:

Eye blink detection (2023)

@article{blinklinmult-fodor23,
  title   = {BlinkLinMulT: Transformer-based Eye Blink Detection},
  author  = {Fodor, {\'A}d{\'a}m and Fenech, Kristian and L{\H{o}}rincz, Andr{\'a}s},
  journal = {Journal of Imaging},
  pages   = {1--19},
  year    = {2023}
}

Personality and sentiment estimation (2022)

@InProceedings{pmlr-v173-fodor22a,
  title     = {Multimodal Sentiment and Personality Perception Under Speech:
               A Comparison of Transformer-based Architectures},
  author    = {Fodor, {\'A}d{\'a}m and Saboundji, Rachid R. and
               Jacques Junior, Julio C. S. and Escalera, Sergio and
               Gallardo-Pujol, David and L{\H{o}}rincz, Andr{\'a}s},
  booktitle = {Understanding Social Behavior in Dyadic and Small Group Interactions},
  pages     = {218--241},
  year      = {2022},
  volume    = {173},
  series    = {Proceedings of Machine Learning Research},
  publisher = {PMLR},
  url       = {https://proceedings.mlr.press/v173/fodor22a.html}
}

Contact

Ádám Fodor — adamfodor.com · fodorad201@gmail.com