A bunch of transformer implementations
Project description
Transformer Implementations
Transformer Implementations and some examples with them
Implemented:
- Vanilla Transformer
- ViT - Vision Transformers
- DeiT - Data efficient image Transformers
- BERT - Bidirectional Encoder Representations from Transformers
- GPT - Generative Pre-trained Transformer
Installation
$ pip install transformer-implementations
or
python setup.py build
python setup.py install
Example
In notebooks directory there is a notebook on how to use each of these models for their intented use; such as image classification for Vision Transformer (ViT) and others. Check them out!
from transformer_package.models import ViT
image_size = 28 # Model Parameters
channel_size = 1
patch_size = 7
embed_size = 512
num_heads = 8
classes = 10
num_layers = 3
hidden_size = 256
dropout = 0.2
model = ViT(image_size,
channel_size,
patch_size,
embed_size,
num_heads,
classes,
num_layers,
hidden_size,
dropout=dropout).to(DEVICE)
prediction = model(image_tensor)
Language Translation
from "Attention is All You Need": https://arxiv.org/pdf/1706.03762.pdf
Models trained with Implementation:
Multi-class Image Classification with Vision Transformers (ViT)
from "An Image is Worth 16x16 words: Transformers for image recognition at scale": https://arxiv.org/pdf/2010.11929v1.pdf
Models trained with Implementation:
Note: ViT will not perform great on small datasets
Multi-class Image Classification with Data-efficient image Transformers (DeiT)
from "Training data-efficient image transformers & distillation through attention": https://arxiv.org/pdf/2012.12877v1.pdf
Models trained with Implementation:
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for transformer_implementations-0.0.8.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9fedfa8f03903ecef0b7980e70c37459d4cc9095640220ee3296939b67337948 |
|
MD5 | 6f64e474f2d07499a1d759bec923d2f4 |
|
BLAKE2b-256 | 15e7d5c72e82f721ba3ea9f6553a5b4b4912d334edcd7b4813981dcf23b96fa2 |
Hashes for transformer_implementations-0.0.8-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7096ccecd2c846613209e9ca0e1fbbd25e2fbbfd6c9d64bf140c48b30c16f75c |
|
MD5 | 02ebb5ef6769b86fc58322021f4db58e |
|
BLAKE2b-256 | a8bb9b9ca146f7ef3c9c97554626aeca1bb94c092f188f9dedcc69a4c4f98bd9 |