atc-torch - Pytorch
Project description
Attention + Convolution transformer
This is an experimental architecture leveraging convolution blocks with attention blocks to model both the short and long range dynamics of the input tokens. The flow is the following: x -> convolution block -> attn -> FFN
Install
``
Usage
import torch
from attnconv.main import ATCTransformer
model = ATCTransformer(
dim=512,
depth=6,
num_tokens=20000,
dim_head=64,
heads=8,
ff_mult=4,
)
x = torch.randint(0, 20000, (1, 512))
logits = model(x) # (1, 1024, 20000)
print(logits)
License
MIT
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
atc_torch-0.0.1.tar.gz
(5.0 kB
view hashes)
Built Distribution
Close
Hashes for atc_torch-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b84682bae3c6a32da53b041c12f2d976d1f33e662a5b9c412357df257ff3a318 |
|
MD5 | ef517e6e7df5c9c5321bd9f65105930e |
|
BLAKE2b-256 | 0119e9c5d11197e894623d62ebfb93232e5d08e77ca6a93d12bb5a95adf179c1 |