bio-attention
Simple implementations of attention modules adapted for the biological data domain.
Why use this package?
There are already plenty of excellent implementations out there that allow you to test out the countless variants of transformers [1], [2]. This repository primarily separates itself from the previous in that it contains positional encodings schemes adapted to allow for irregularly-spaced positions in sequences. This is useful in, for example: (1) the mass spectral domain (proteomics, metabolomics, ...), where transformers operate on sets of peaks, (2) any kind of (epi)genomic data that measures sites of interests on the genome that are irregularly-spaced (such as WGBS/CpG sites, ATAC-seq/chromatin accessibility, ...). Additionally, the attention definitions in this repository are compatible with multi-dimensional data, such as the MSAs used in some protein language models, and AlphaFold.
Install
Since PyTorch is a dependency of bio-attention, we recommend installing PyTorch independently first, as your system may require a specific version (e.g. CUDA drivers).
After PyTorch installation, bio-attention can be installed using pip
pip install bio-attention
Note
This package used to be a 2D sliding window attention package. The current formulation of the package does not allow for this type of attention anymore (instead, I recommend to perform axial attention with alternating sliding window attention across one axis and full self-attention across the other). If you want to use 2D sliding window attention, check out the old version of this repo.
Usage
Package roadmap
- Embedding layers
- Continuous
- Discrete
- Binary
- Bin
- [~] Positional encoding schemes
- Sinusoidal
- Embedding
- Continuous
- Rotary
- AliBi
- DPB
- XL
- Test support for multi-dimensional inputs
- [~] Attention modules
- Vanilla
- Windowed
- Random
- Performer
- Encoder
- Decoder
- Cross
- Support for multi-dim inputs
- Add a warning if non-increasing positional indices are used with a decoder attention
- Add docs clarifying that clf tokens are automatically accounted for if no pos is provided for them
- Tests
- Typing
- Docs
1 maintainer
Release files for bio-attention 0.1.15
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| bio_attention-0.1.15.tar.gz | 2.2 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| bio_attention-0.1.15-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 2.2 MB
Release files / bio_attention-0.1.15.tar.gz
| Download URL | bio_attention-0.1.15.tar.gz |
|---|---|
| Size | 2.2 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c69ba70e0e6e264f770128100c7e9a7eddf5b3808887da41018de59dc0a5b23f
|
|
BLAKE2b-256 checksum How to use checksums |
d22fe259dee5db7eb374e23c10118c9b981ce2fcbb44d093aa04f7ba764df8e9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Release files / bio_attention-0.1.15-py3-none-any.whl
| Download URL | bio_attention-0.1.15-py3-none-any.whl |
|---|---|
| Size | 17.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
62a88f5e776f75c2cac122fa739117a87bf9afb0fdfde933daf250d5a3c2767c
|
|
BLAKE2b-256 checksum How to use checksums |
0971ca7099f37d02c4fcf74c178ec7c871608272ab4bca431c072c3cbd354e6e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
