Multi-Modal Casual Multi-Grouped Query Attention
Experiments around using Multi-Modal Casual Attention with Multi-Grouped Query Attention
Appreciation
- Lucidrains
- Agorians
Install
pip install mmmgqa
Usage
import torch
from mmca_mgqa.attention import SimpleMMCA
# Define the dimensions
dim = 512
head = 8
seq_len = 10
batch_size = 32
#attn
attn = SimpleMMCA(dim=dim, heads=head)
#random tokens
v = torch.randn(batch_size, seq_len, dim)
t = torch.randn(batch_size, seq_len, dim)
#pass the tokens throught attn
tokens = attn(v, t)
print(tokens)
Architecture
Todo
License
MIT
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
mmmgqa-0.0.2.tar.gz
(3.0 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file mmmgqa-0.0.2.tar.gz.
File metadata
- Download URL: mmmgqa-0.0.2.tar.gz
- Upload date:
- Size: 3.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
poetry/1.3.2 CPython/3.11.0 Darwin/22.4.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
0ab9320fe3693b5590a560966c6460c96d36748e5e3bab8d3ca3fa69581ffcc0
|
|
| MD5 |
620af15dd76899d136479189bb29f375
|
|
| BLAKE2b-256 |
e28f6a39928e215f056514b984d4a559a194e0504a9782205a67f0cd1b319d6a
|
File details
Details for the file mmmgqa-0.0.2-py3-none-any.whl.
File metadata
- Download URL: mmmgqa-0.0.2-py3-none-any.whl
- Upload date:
- Size: 3.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
poetry/1.3.2 CPython/3.11.0 Darwin/22.4.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
838f2005dfd24f9357024b63c2d405490929ca23c216398076cf54db4b882769
|
|
| MD5 |
7f02092a199aec00f4baeacb0e76bf21
|
|
| BLAKE2b-256 |
3f867d45f22fb76b40649a22ddd8c0ac322cb8f821a2f17fd0670a93947e67f5
|
