TRL - Transformers Reinforcement Learning
A comprehensive library to post-train foundation models
🎉 What's New
🔥 Muse Spark Open: Fine-tune Muse Spark Open with SFTTrainer, sharded across your GPUs at load time. See the 8-node LoRA example.
🚀 Training on AWS Trainium: TRL's trainers now run on AWS Trainium through TorchNeuron, PyTorch's native Neuron backend.
Overview
TRL is a cutting-edge library designed for post-training foundation models using advanced techniques like Supervised Fine-Tuning (SFT), Group Relative Policy Optimization (GRPO), and Direct Preference Optimization (DPO). Built on top of the 🤗 Transformers ecosystem, TRL supports a variety of model architectures and modalities, and can be scaled-up across various hardware setups.
Highlights
-
Trainers: Various fine-tuning methods are easily accessible via trainers like
SFTTrainer,GRPOTrainer,DPOTrainer,KTOTrainerand more. -
Efficient and scalable:
- Leverages 🤗 Accelerate to scale from single GPU to multi-node clusters using methods like DDP and DeepSpeed.
- Full integration with 🤗 PEFT enables training on large models with modest hardware via quantization and LoRA/QLoRA.
- Integrates 🦥 Unsloth for accelerating training using optimized kernels.
-
Command Line Interface (CLI): A simple interface lets you fine-tune with models without needing to write code.
Installation
Python Package
Install the library using pip:
pip install trl
From source
If you want to use the latest features before an official release, you can install TRL from source:
pip install git+https://github.com/huggingface/trl.git
Repository
If you want to use the examples you can clone the repository with the following command:
git clone https://github.com/huggingface/trl.git
Quick Start
For more flexibility and control over training, TRL provides dedicated trainer classes to post-train language models or PEFT adapters on a custom dataset. Each trainer in TRL is a light wrapper around the 🤗 Transformers trainer and natively supports distributed training methods like DDP, DeepSpeed ZeRO, and FSDP.
SFTTrainer
Here is a basic example of how to use the SFTTrainer:
from trl import SFTTrainer
from datasets import load_dataset
dataset = load_dataset("trl-lib/Capybara", split="train")
trainer = SFTTrainer(
model="Qwen/Qwen2.5-0.5B",
train_dataset=dataset,
)
trainer.train()
GRPOTrainer
GRPOTrainer implements the Group Relative Policy Optimization (GRPO) algorithm that is more memory-efficient than PPO and was used to train Deepseek AI's R1.
from datasets import load_dataset
from trl import GRPOTrainer
from trl.rewards import accuracy_reward
dataset = load_dataset("trl-lib/DeepMath-103K", split="train")
trainer = GRPOTrainer(
model="Qwen/Qwen2.5-0.5B-Instruct",
reward_funcs=accuracy_reward,
train_dataset=dataset,
)
trainer.train()
DPOTrainer
DPOTrainer implements the popular Direct Preference Optimization (DPO) algorithm that was used to post-train Llama 3 and many other models. Here is a basic example of how to use the DPOTrainer:
from datasets import load_dataset
from trl import DPOTrainer
dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
trainer = DPOTrainer(
model="Qwen/Qwen3-0.6B",
train_dataset=dataset,
)
trainer.train()
KTOTrainer
KTOTrainer implements the Kahneman-Tversky Optimization (KTO) algorithm, which aligns models from simple binary (desirable / undesirable) feedback rather than paired preferences. Here is a basic example of how to use the KTOTrainer:
from datasets import load_dataset
from trl import KTOTrainer
dataset = load_dataset("trl-lib/kto-mix-14k", split="train")
trainer = KTOTrainer(
model="Qwen/Qwen3-0.6B",
train_dataset=dataset,
)
trainer.train()
RewardTrainer
Here is a basic example of how to use the RewardTrainer:
from trl import RewardTrainer
from datasets import load_dataset
dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
trainer = RewardTrainer(
model="Qwen/Qwen2.5-0.5B-Instruct",
train_dataset=dataset,
)
trainer.train()
Command Line Interface (CLI)
You can use the TRL Command Line Interface (CLI) to quickly get started with post-training methods like Supervised Fine-Tuning (SFT) or Direct Preference Optimization (DPO):
SFT:
trl sft --model_name_or_path Qwen/Qwen2.5-0.5B \
--dataset_name trl-lib/Capybara \
--output_dir Qwen2.5-0.5B-SFT
DPO:
trl dpo --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
--dataset_name argilla/Capybara-Preferences \
--output_dir Qwen2.5-0.5B-DPO
KTO:
trl kto --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
--dataset_name trl-lib/kto-mix-14k \
--output_dir Qwen2.5-0.5B-KTO
Read more about CLI in the relevant documentation section or use --help for more details.
Development
If you want to contribute to trl or customize it to your needs make sure to read the contribution guide and make sure you make a dev install:
git clone https://github.com/huggingface/trl.git
cd trl/
pip install -e .[dev]
Experimental
A minimal incubation area is available under trl.experimental for unstable / fast-evolving features. Anything there may change or be removed in any release without notice.
Example:
from trl.experimental.new_trainer import NewTrainer
Read more in the Experimental docs.
Citation
@software{vonwerra2020trl,
title = {{TRL: Transformers Reinforcement Learning}},
author = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
license = {Apache-2.0},
url = {https://github.com/huggingface/trl},
year = {2020}
}
License
This repository's source code is available under the Apache-2.0 License.
5 maintainers
Metadata
5 maintainers
Release files for trl 1.16.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| trl-1.16.0.tar.gz | 774.7 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| trl-1.16.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 1.7 MB
Release files / trl-1.16.0.tar.gz
| Download URL | trl-1.16.0.tar.gz |
|---|---|
| Size | 774.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
999bd4a85b08870b7bfa387e1efa5177d2a4b88fe82b2526a057ac5d3cc600cd
|
|
BLAKE2b-256 checksum How to use checksums |
72332cce5d6c5398d6de2dad22afa713f1a65569910685ab3598d1767d4d8483
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / trl-1.16.0-py3-none-any.whl
| Download URL | trl-1.16.0-py3-none-any.whl |
|---|---|
| Size | 948.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
c4726e139dd40e1a41661511c639c8a68416782b2e3e8c0befebd77bb7ba0396
|
|
BLAKE2b-256 checksum How to use checksums |
e14dc5eaed26c16bfafc2f724e6be320f94a4f26bf4bb4ba62917114dd9b60c8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
