Attention-and-Transformers 0.0.9

Building attention mechanisms and Transformer models from scratch. Alias ATF.

Attention mechanisms and Transformers

• This goal of this repository is to host basic architecture and model traning code associated with the different attention mechanisms and transformer architecture.
• At the moment, I more interested in learning and recreating these new architectures from scratch than full-fledged training. For now, I'll just be training these models on small datasets.

Installation

• Using pip to install from pypi

pip install Attention-and-Transformers

• Using pip to install latest version from github

pip install git+https://github.com/veb-101/Attention-and-Transformers.git

• Local clone and install

git clone https://github.com/veb-101/Attention-and-Transformers.git atf
cd atf
python setup.py install

Test Installation

python load_test.py

Attention Mechanisms

# No.	Mechanism	Paper
1	Multi-head Self Attention	Attention is all you need
2	Multi-head Self Attention 2D	MobileViT V1
2	Separable Self Attention	MobileViT V2

Transformer Models

# No.	Models	Paper
1	Vision Transformer	An Image is Worth 16x16 Words:
2	MobileViT-V1	MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer
3	MobileViT-V2 (under development)	Separable Self-attention for Mobile Vision Transformers