Convenient Text-to-Text Training for Transformers
Project description
t2t-tuner
Convenient Text-to-Text Training for Transformers
pip install t2t-tuner
Requires PyTorch: either follow PyTorch installation instructions or use a PyTorch container.
Features
- Easy training for text-to-text generation tasks
- Training methods/features:
- Supervised fine-tuning
- Gradient checkpointing
- Model parallelism
- Soft prompt tuning (based on this paper)
- Freeze encoder/decoder/embeddings
- Print model summary
- Based on the wonderful HuggingFace Transformers library. Tested on T5-based models. In theory, it should work with other models that support AutoModelForSeq2SeqLM as well
This work is based on HuggingFace's run_translation.py script for text-to-text generation tasks. It provides (what I feel is) a more convenient interface to training and inferencing text-to-text generation models, along with better access to some features and new features that I added in myself.
Examples
Simple snippet:
import t2t
trainer_arguments = t2t.TrainerArguments(model_name_or_path="t5-small",
train_file=YOUR_DATASET)
trainer = t2t.Trainer(arguments=trainer_arguments)
# train without validation
trainer.train(valid=False)
For more concrete examples, check out the notebooks linked below:
Development
Building Package
python3 -m pip install --upgrade build twine
python3 -m build
python3 -m twine upload dist/*
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
t2t-tuner-0.0.6.tar.gz
(8.3 kB
view hashes)
Built Distribution
Close
Hashes for t2t_tuner-0.0.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4f0bb880ffbcb3661591bd56cb0820e170e25bc2b2c12e4114690a256c436d00 |
|
MD5 | 9e6a8d4a5ea0b17b816334317ddda2ae |
|
BLAKE2b-256 | 63dd609864ac4f047342a9bc41d813079bae9054cb4be88e0c519b26e0322089 |