keras-transformer-xl 0.2.1

Transformer-XL implemented in Keras

Keras Transformer-XL

[中文|English]

Unofficial implementation of Transformer-XL.

Install

pip install keras-transformer-xl

Usage

Load Pretrained Weights

Several configuration files can be found at the info directory.

import os
from keras_transformer_xl import load_trained_model_from_checkpoint

checkpoint_path = 'foo/bar/sota/enwiki8'
model = load_trained_model_from_checkpoint(
    config_path=os.path.join(checkpoint_path, 'config.json'),
    checkpoint_path=os.path.join(checkpoint_path, 'model.ckpt')
)
model.summary()

About IO

Suppose the number of transformer blocks is n. The last n inputs are used for inputs of memorization, and the last n outputs represents new data to be memorized.

Use tensorflow.python.keras

Add TF_KERAS=1 to environment variables to use tensorflow.python.keras.