Attention-and-Transformers 0.0.1

Building attention mechanisms and Transformer models from scratch. Alias ATF. https://github.com/veb-101/Attention-and-Transformers

Attention mechanisms and Transformers

• This goal of this repository is to host basic architecture and model traning code associated with the different attention mechanisms and transformer architecture.

• At the moment, I more interested in learning and recreating these new architectures from scratch than full-fledged training. For now, I'll just be training these models on small datasets.

Attention Mechanisms

# No.	Mechanism	Paper
1	Multi-head Self Attention	Attention is all you need
2	Multi-head Self Attention 2D	MobileViT V1

Transformer Models

# No.	Models	Paper
1	Vision Transformer	An Image is Worth 16x16 Words:
2	MobileViT-V1	MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer
3	MobileViT-V2 (under development)	Separable Self-attention for Mobile Vision Transformers