transformer-implementations 0.0.6

A bunch of transformer implementations

Transformer Implementations

Transformer Implementations and some examples with them

Implemented:

• Vanilla Transformer
• ViT - Vision Transformers
• DeiT - Data efficient image Transformers

Installation

PyPi Installation

$ pip install transformer-implementations

Language Translation

from "Attention is All You Need": https://arxiv.org/pdf/1706.03762.pdf

Models trained with Implementation:

• Multi30k - German to English

Multi-class Image Classification with Vision Transformers (ViT)

from "An Image is Worth 16x16 words: Transformers for image recognition at scale": https://arxiv.org/pdf/2010.11929v1.pdf

Models trained with Implementation:

• MNIST - Grayscale Images
• CIFAR10 - MultiChannel Images

Multi-class Image Classification with Data-efficient image Transformers (DeiT)

from "Training data-efficient image transformers & distillation through attention": https://arxiv.org/pdf/2012.12877v1.pdf

Models trained with Implementation:

• Pretraining Teacher model for Distillation
• CIFAR10 - Low Res Images