Attention on Attention - Pytorch
Project description
Attention on Attention Implementation
This is a practice implementation after randomly finding it on Lucidrain's repo, I'm implementing the model architecture just for practice!
Basically the architecture is: x => q, k, v -> multihead attn with residual q -> concat -> 2 linear projects ->sigmoid -> mult -> add -> norm -> ffn -> add -> norm with residual of first add and norm
Install
pip3 install
Usage
AoA
Module
import torch
from aoa.main import AoA
x = torch.randn(1, 10, 512)
model = AoA(512, 8, 64, 0.1)
out = model(x)
print(out.shape)
AoATransformer
import torch
from aoa.main import AoATransformer
x = torch.randint(0, 100, (1, 10))
model = AoATransformer(512, 1, 100)
out = model(x)
print(out.shape)
Citations
@misc{rahman2020improved,
title = {An Improved Attention for Visual Question Answering},
author = {Tanzila Rahman and Shih-Han Chou and Leonid Sigal and Giuseppe Carenini},
year = {2020},
eprint = {2011.02164},
archivePrefix = {arXiv},
primaryClass = {cs.CV}
}
@misc{huang2019attention,
title = {Attention on Attention for Image Captioning},
author = {Lun Huang and Wenmin Wang and Jie Chen and Xiao-Yong Wei},
year = {2019},
eprint = {1908.06954},
archivePrefix = {arXiv},
primaryClass = {cs.CV}
}
License
MIT
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
aoa_torch-0.0.1.tar.gz
(4.8 kB
view hashes)
Built Distribution
Close
Hashes for aoa_torch-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 54b1bbb283c0808c31443a4cb8a7dedbd77273429038b909d997ae615b05f2e8 |
|
MD5 | 95513fa308bb4e54f9c7e7fe0cb83a4d |
|
BLAKE2b-256 | 38ee15ce1d0846007dc5a59da20bdd6df699dff42e55ad6cb033b186af6dfb7d |