zeldarose 0.3.0

Train transfomer-based models

Zelda Rose

A trainer for transformer-based models.

Installation

Simply install with pip (preferably in a virtual env, you know the drill)

pip install zeldarose

Train a model

Here is a short example:

zeldarose-tokenizer --vocab-size 4096 --out-path local/tokenizer  --model-name "my-muppet" tests/fixtures/raw.txt
zeldarose-transformer --tokenizer local/tokenizer --pretrained-model flaubert/flaubert_small_cased --out-dir local/muppet --val-text tests/fixtures/raw.txt tests/fixtures/raw.txt

There are other parameters (see zeldarose-transformer --help for a comprehensive list), the one you are probably mostly interested in is --config (for which there is an example target in examples/).

The parameters --pretrained-models, --tokenizer and --model-config are all fed directly to Huggingface's transformers and can be pretrained models names or local path.

Distributed training

This is somewhat tricky, you have several options

• If you are running in a SLURM cluster use --accelerator ddp and invoke via srun

• Otherwise you have two options

  • Run with --accelerator ddp_spawn, which uses multiprocessing.spawn to start the process swarm (tested, but possibly slower and more limited, see pytorch-lightning doc)
  • Run with --accelerator ddp and start with torch.distributed.launch with --use_env and --no_python (untested)

Whatever you do, for now it's safer to run once without distributed training in order to preprocess the raw texts in a predictable environment.

Inspirations

• https://github.com/shoarora/lmtuners
• https://github.com/huggingface/transformers/blob/243e687be6cd701722cce050005a2181e78a08a8/examples/run_language_modeling.py