simplepe 0.0.3

A simple simplepe for positional encoding

SimplePE

Simple implementation of some Positional Encoding methods in PyTorch.

• Sinusoidal Positional Encoding
• Rotary Positional Encoding

Installation

git clone https://github.com/donglinkang2021/simplepe.git
cd simplepe
pip install .

Usage

python example.py # pe value
python example2.py # attention score

You can see the following results:

Method	Sinusoidal	Rotary
Dimension-Time_Step Heatmap
Value-Time_Step Line_Plot
Multi-Head Dimension-Time_Step Heatmap
Multi-Head Value-Time_Step Line_Plot

Method\Time_Step-Time_Step Heatmap	T=2048, D=32	T=2048, D=64	T=4096, D=512
Sinusoidal
Rotary

Method\Attention_Score-Time_Step Line_Plot	T=2048, D=32	T=2048, D=64	T=4096, D=512
Sinusoidal
Rotary