transformer-lib 0.1.0

Attention mechanisms from Every_Thing_About_Transformer, as an importable library

transformer-lib

Attention mechanisms from Every_Thing_About_Transformer, as an importable Python library.

Installation

pip install -e .

Usage

from transformer_lib import MultiHeadAttention
import torch

mha = MultiHeadAttention(embed_dim=512, num_heads=8)
x = torch.randn(2, 10, 512)
out = mha(x, x, x)

Available Modules

Module	Description
SelfAttention	Basic scaled dot-product self-attention
MultiHeadAttention	Multi-head attention with separate Q/K/V projections
CausalAttention	Multi-head with lower-triangular causal mask
CrossAttention	Encoder-decoder cross-attention with self-attention pre-step
GlobalAttention	Additive (Bahdanau-style) alignment scoring
MultiQueryAttention	Shared K/V across all heads
GroupedQueryAttention	Groups of query heads sharing K/V (GQA)
MultiHeadLatentAttention	Latent-space compression for keys/values