Multi-Modal Casual Multi-Grouped Query Attention
Experiments around using Multi-Modal Casual Attention with Multi-Grouped Query Attention
Appreciation
- Lucidrains
- Agorians
Install
pip install mmmgqa
Usage
import torch
from mmca_mgqa.attention import SimpleMMCA
# Define the dimensions
dim = 512
head = 8
seq_len = 10
batch_size = 32
#attn
attn = SimpleMMCA(dim=dim, heads=head)
#random tokens
v = torch.randn(batch_size, seq_len, dim)
t = torch.randn(batch_size, seq_len, dim)
#pass the tokens throught attn
tokens = attn(v, t)
print(tokens)
Architecture
Todo
License
MIT
1 maintainer
Release files for mmmgqa 0.0.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| mmmgqa-0.0.2.tar.gz | 3.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| mmmgqa-0.0.2-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 6.0 kB
Release files / mmmgqa-0.0.2.tar.gz
| Download URL | mmmgqa-0.0.2.tar.gz |
|---|---|
| Size | 3.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0ab9320fe3693b5590a560966c6460c96d36748e5e3bab8d3ca3fa69581ffcc0
|
|
BLAKE2b-256 checksum How to use checksums |
e28f6a39928e215f056514b984d4a559a194e0504a9782205a67f0cd1b319d6a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.3.2 CPython/3.11.0 Darwin/22.4.0
|
Release files / mmmgqa-0.0.2-py3-none-any.whl
| Download URL | mmmgqa-0.0.2-py3-none-any.whl |
|---|---|
| Size | 3.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
838f2005dfd24f9357024b63c2d405490929ca23c216398076cf54db4b882769
|
|
BLAKE2b-256 checksum How to use checksums |
3f867d45f22fb76b40649a22ddd8c0ac322cb8f821a2f17fd0670a93947e67f5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.3.2 CPython/3.11.0 Darwin/22.4.0
|
