t2t-tuner 0.0.4

Convenient Text-to-Text Training for Transformers

t2t-tuner

Convenient Text-to-Text Training for Transformers

pip install t2t-tuner

Requires PyTorch: either follow PyTorch installation instructions or use a PyTorch container.

Features

• Easy training for text-to-text generation tasks
• Training methods/features:
  • Supervised fine-tuning
  • Gradient checkpointing
  • Model parallelism
  • Soft prompt tuning (based on this paper)
  • Freeze encoder/decoder/embeddings
  • Print model summary
• Based on the wonderful HuggingFace Transformers library. Tested on T5-based models. In theory, it should work with other models that support AutoModelForSeq2SeqLM as well

This work is based on HuggingFace's run_translation.py script for text-to-text generation tasks. It provides (what I feel is) a more convenient interface to training and inferencing text-to-text generation models, along with better access to some features and new features that I added in myself.

Examples

Simple snippet:

import t2t

trainer_arguments = t2t.TrainerArguments(model_name_or_path="t5-small",
                                         train_file=YOUR_DATASET)

trainer = t2t.Trainer(arguments=trainer_arguments)

trainer.train()

For more concrete examples, check out the notebooks linked below:

• Simple example
• Soft Prompt Tuning

Development

Building Package

python3 -m pip install --upgrade build twine
python3 -m build
python3 -m twine upload dist/*