keras-self-attention 0.0.11

Attention mechanism for processing sequence data that considers the context for each timestamp

Attention mechanism for processing sequence data that considers the context for each timestamp.

• 
• 
• 
• 

Install

pip install keras-self-attention

Usage

import keras
from keras_self_attention import Attention


model = keras.models.Sequential()
model.add(keras.layers.Embedding(input_dim=10000,
                                 output_dim=300,
                                 mask_zero=True))
model.add(keras.layers.Bidirectional(keras.layers.LSTM(units=128,
                                                       return_sequences=True)))
model.add(Attention())
model.add(keras.layers.Dense(units=5))
model.compile(
    optimizer='adam',
    loss='categorical_crossentropy',
    metrics=['categorical_accuracy'],
)
model.summary()

attention_width

The global context may be too broad for one piece of data. The parameter attention_width controls the width of the local context.

attention_activation

The activation function of e_{t, t'}. There is no activation by default.

from keras_self_attention import Attention

Attention(
    attention_width=15,
    attention_activation='sigmoid',
    name='Attention',
)