⚡ FlashRL ⚡

Fast RL training with Quantized Rollouts (Blog)

What is FlashRL? • Quick Start • Usage Guide • Examples • Road Map • Citation

FlashRL patches the inference backend to generate RL rollouts in INT8 & FP8, with accurate rollout logprob.

Figure 1. Left: AIME accuracy of Qwen2.5-32B DAPO training with INT8 and BF16 precisions for rollout generation using vLLM engine. Right: Training throughput (updates per hour) in the DAPO training (vLLM + BF16 FSDP).

⚡ Quick Start

Installation

pip install flash-llm-rl # need to be installed in all nodes in multi-node training

(Optional) there are two options to verify the FlashRL install: 1) set FLASHRL_LOGGING_LEVEL to DEBUG and compare the log with the provided ones; 2) for more details / debugging, please follow the Tutorial.

Rollout Generation w. FP8 Quantization

FlashRL is implemented as a plug-in-and-play manner, using environment variables FLASHRL_CONFIG to control the quantization precision. Note that, due to quantization overhead, it is only recommended to use quantized rollout in the combination of large model (i.e., 14B+, preferrable 32B+) and long cot generation (DAPO training instead of GSM8K training).

# for single-node job
export FLASHRL_CONFIG=fp8
bash verl/examples/ppo_trainer/run_qwen2.5-32b.sh

# alternatively, for multi-node jobs via `ray submit`, fp8 online quantization will be turned on via
# > echo "  FLASHRL_CONFIG: 'fp8'" | tee -a verl/trainer/runtime_env.yaml # add `FLASHRL_CONFIG: 'fp8'` to runtime env
# > bash verl/recipe/dapo/run_dapo_qwen2.5_32b.sh # this can be any scripts

RL Logprob Patch Only

Setting the config to bf16 to extract precise logprob used in sampling without rollout quantization. This is useful for applying the Truncated Importance Sampling.

#  for single-node job
export FLASHRL_CONFIG=bf16
bash verl/examples/ppo_trainer/run_qwen2.5-32b.sh

# alternatively, for multi-node jobs via `ray submit`, RL Logprob Patch Only will be turned on via
# > echo "  FLASHRL_CONFIG: 'bf16'" | tee -a verl/trainer/runtime_env.yaml # add `FLASHRL_CONFIG: 'fp8'` to runtime env
# > bash verl/recipe/dapo/run_dapo_qwen2.5_32b.sh # this can be any scripts

Usage Guide

FlashRL has 3 major functionality, profiling, configure helper, and patcher.

Profiling (optional for fp8 and bf16)

This step is not needed for the native fp8 online quantization supported by vLLM, and the logprog-only path bf16, and is needed for int8 or fp8_channel quantization. Specifically, profilling compares a bf16 model and a quantized model to decide how the online quantization should be performed for an updated model. Please find below an example for Qwen/Qwen2.5-32B and Qwen/Qwen2.5-0.5B-Instruct. The quantized model can be any w8a8/fp8 model produced by llm-compressor. Note that, Redhat AI provides various quantized models, and can be used here as in the 0.5B-Instruct example.

# for `Qwen/Qwen2.5-32B`
flashrl profile -m Qwen/Qwen2.5-32B -qm LiyuanLucasLiu/Qwen2.5-32B-quantized.w8a8 -o ${PROFILE_PATH:-"$HOME/profile.32b.pt"} --fn int8

# for `Qwen/Qwen2.5-0.5B-Instruct`
flashrl profile -m Qwen/Qwen2.5-0.5B-Instruct -qm RedHatAI/Qwen2.5-0.5B-Instruct-quantized.w8a8 -o ${PROFILE_PATH:-"$HOME/profile.0_5b.pt"} --fn int8

Configure Helper (optional for fp8 and bf16)

This step is not needed for the native fp8 online quantization supported by vLLM, and the logprog-only path bf16, and is needed for int8 or fp8_channel quantization. Specifically, configure helper creates a yaml file for the patcher to use. Please find below an example for Qwen/Qwen2.5-32B and Qwen/Qwen2.5-0.5B-Instruct.

# for `Qwen/Qwen2.5-32B`
flashrl setup -m LiyuanLucasLiu/Qwen2.5-32B-quantized.w8a8 -p $HOME/profile.32b.pt --fn int8 -o ${CONFIG_PATH:-"$HOME/.flashrl_config.32b.yaml"}

# for `Qwen/Qwen2.5-0.5B-Instruct`
flashrl setup -m RedHatAI/Qwen2.5-0.5B-Instruct-quantized.w8a8 -p $HOME/profile.0_5b.pt --fn int8 -o ${CONFIG_PATH:-"$HOME/.flashrl_config.0_5b.yaml"}

Patcher

Patcher would check the environment variable and operates accordingly. Please find the supported environment variables as below.

Examples

Tested Environments

Below are the combinations of the environments that we have tested on.

🚧 Roadmap & Future Improvements

We're working on several improvements to Flash-RL:

• Support of Other RL Toolkits: Currently Flash-RL only supports VeRL, we are working on rolloing out support for other packages like OpenRLHF
• Support of Other LLM Inference Toolkits: Currently Flash-RL only supports vLLM, we are working on rolloing out support for other tollkits like SgLang
• Further Throughput Optimization: We are working on implementing efficient GPU kernels to accelerate online quantization

📚 Citation

If you find our work useful, please cite us:

@misc{yao2025offpolicy,
  title = {Your Efficient RL Framework Secretly Brings You Off-Policy RL Training},
  url = {https://fengyao.notion.site/off-policy-rl},
  author = {Yao, Feng and Liu, Liyuan and Zhang, Dinghuai and Dong, Chengyu and Shang, Jingbo and Gao, Jianfeng},
  journal = {Feng Yao's Notion},
  year = {2025},
  month = aug,
}
@misc{yao2025flashrl,
  title = {FlashRL: 8Bit Rollouts, Full Power RL},
  url = {https://fengyao.notion.site/flash-rl},
  author = {Liu, Liyuan and Yao, Feng and Zhang, Dinghuai and Dong, Chengyu and Shang, Jingbo and Gao, Jianfeng},
  journal = {Feng Yao's Notion},
  year = {2025},
  month = aug,
}

Questions?

If you have any questions related to the code or the blog, feel free to reach out to us at Liyuan Liu and Feng Yao.