simplicial-attention

Simplicial Attention

Implementation of 2-simplicial attention proposed by Clift et al. (2019) and the recent attempt to make practical in Fast and Simplex, Roy et al. (2025)

Paper explanation by Gabriel Mongaras

Appreciation

Tejas for finding my error in the Triton backwards kernel!

Install

$ pip install simplicial-attention

Usage

import torch
from simplicial_attention.triton_two_simplicial_attention import SlidingWindowTwoSimplicialMHA

higher_order_attn = SlidingWindowTwoSimplicialMHA(
    dim = 512,
    dim_head = 64,
    heads = 8
).cuda()

tokens = torch.randn(2, 1024, 512).cuda()

assert higher_order_attn(tokens).shape == tokens.shape

Example

Enwik8, every 2 layers

$ pip install '.[examples]' && python train.py

Contributing

First install with pytest

$ pip install '.[test]'

Then add your code and make sure it passes

$ pytest tests

Citations

@misc{roy2025fastsimplex2simplicialattention,
    title   = {Fast and Simplex: 2-Simplicial Attention in Triton}, 
    author  = {Aurko Roy and Timothy Chou and Sai Surya Duvvuri and Sijia Chen and Jiecao Yu and Xiaodong Wang and Manzil Zaheer and Rohan Anil},
    year    = {2025},
    eprint  = {2507.02754},
    archivePrefix = {arXiv},
    primaryClass = {cs.LG},
    url     = {https://arxiv.org/abs/2507.02754}, 
}

@misc{clift2019logic2simplicialtransformer,
    title   = {Logic and the $2$-Simplicial Transformer}, 
    author  = {James Clift and Dmitry Doryn and Daniel Murfet and James Wallbridge},
    year    = {2019},
    eprint  = {1909.00668},
    archivePrefix = {arXiv},
    primaryClass = {cs.LG},
    url     = {https://arxiv.org/abs/1909.00668}, 
}

@article{Peng2024OnLO,
    title     = {On Limitations of the Transformer Architecture},
    author    = {Binghui Peng and Srini Narayanan and Christos Papadimitriou},
    journal   = {ArXiv},
    year      = {2024},
    volume    = {abs/2402.08164},
    url       = {https://api.semanticscholar.org/CorpusID:267636545}
}

Metadata

Release files for simplicial-attention 0.1.6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for simplicial-attention 0.1.6
File	Size	Uploaded
simplicial_attention-0.1.6.tar.gz	37.3 MB	Sep 2, 2025	Details

Built distribution (wheel)

Table of built distributions (wheels) for simplicial-attention 0.1.6
File	Interpreter	ABI	Platform	Reset
simplicial_attention-0.1.6-py3-none-any.whl 16.8 kB Sep 2, 2025	Python 3	none	any	Details

Total release size: 37.3 MB

Release files / simplicial_attention-0.1.6.tar.gz

Download URL	simplicial_attention-0.1.6.tar.gz
Size	37.3 MB
Tags	Source
SHA-256 checksum How to use checksums	`c38d9a6e1fac59ad6902770bee6bc9c8ae82fb2cc04abf6a6bbd0ae614f47c92`
BLAKE2b-256 checksum How to use checksums	`0259a8e6608de312d75016b213dffc8e50a65e5e390c899d0e10b8463dd5ba45`
Upload date	Sep 2, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.9.23`

Release files / simplicial_attention-0.1.6-py3-none-any.whl

Download URL	simplicial_attention-0.1.6-py3-none-any.whl
Size	16.8 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`e70ccc413ec690aec42bb44647b9a463a4117572c461b276ad2b4f1ad0434c67`
BLAKE2b-256 checksum How to use checksums	`1ea55d280292099370a040806115e8eff2a307dbee75c88b9397c608affe539c`
Upload date	Sep 2, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.9.23`