reformer-fastai 0.0.2

Fastai community entry to 2020 Reproducibility Challenge

Reformer Reproducibility Experiments

Fastai community entry to 2020 Reproducibility Challenge

Installation

Setup

If you don't already, its a good idea to install the package into a virtual environment

python3 -m venv my_env

source ./my_env/bin/activate

Run Install

Then you can install the package by running:

pip install git+git://github.com/arampacha/reformer_fastai.git

Project Links

• Older repo
• Repo docs
• Fastai forums thread
• Google doc
• RC2020 Report Style Template

Resources

Author's Code and Resources

• Reformer Paper
• Authors ICLR video
• Google Blog
• Authors code (TRAX)
• Reformer enwik8 model and training config

More Code

• @lucidrain’s Reformer code
• HuggingFace: Reformer source code
  • Reformer enwik8 configs
  • Reformer WMT14 en-de configs
  • Reformer Machine Translation example = SubwordTextEncoder tokenizer used for Machine Translation
• HuggingFace: Reformer notebook example
• HuggingFace: long sequences
• HuggingFace: Pretraining

Data

enwik8

• enwik8.zip, raw data, 100mb
• Tensor2Tensor enwik8 data generator code, with train/dev/test split. File lengths:
  • Train: 89,621,832
  • Eval: 5,000,000
  • Test: 5,000,000
• enwik8 notebook Tensor2Tensor

WMT14

• WMT on HuggingFace Datasets
• Reformer WMT14 vocab
• Reformer.input_vocab_size = 33300, from WMT14 model config
• Train Test split: (guess) newstest2013 for validation and newstest2014 for test, in consistence with Vaswani et al. (2017) - from https://arxiv.org/pdf/2009.02070.pdf
• Tokenizer: Tensor2Tensor SubWordTextEncoder

Explainers

• Yannic K explainer
• HuggingFace blog post
• Illustrating the Reformer blog post
• Reformer Deep Dive

Related

• Coursera Attention Models in NLP course, with Reformer co-author
• @hallvagi Attention blogpost
• The Transformer Family by @lilianweng
• A Family of Attention Mechanisms bu @lilianweng