finetrainers 0.1.0

Finetrainers is a work-in-progress library to support (accessible) training of diffusion models

finetrainers 🧪

Finetrainers is a work-in-progress library to support (accessible) training of diffusion models. Our first priority is to support LoRA training for all popular video models in Diffusers, and eventually other methods like controlnets, control-loras, distillation, etc.

Your browser does not support the video tag.	Your browser does not support the video tag.
CogVideoX LoRA training as the first iteration of this project	Replication of PikaEffects

Table of Contents

• Quickstart
• Features
• News
• Support Matrix
• Featured Projects
• Acknowledgements

Quickstart

Clone the repository and make sure the requirements are installed: pip install -r requirements.txt and install diffusers from source by pip install git+https://github.com/huggingface/diffusers. The requirements specify diffusers>=0.32.1, but it is always recommended to use the main branch of Diffusers for the latest features and bugfixes. Note that the main branch for finetrainers is also the development branch, and stable support should be expected from the release tags.

Checkout to the latest stable release tag:

git fetch --all --tags
git checkout tags/v0.1.0

Follow the instructions mentioned in the README for the latest stable release.

Using the main branch

To get started quickly with example training scripts on the main development branch, refer to the following:

• LTX-Video Pika Effects Crush
• CogVideoX Pika Effects Crush
• Wan T2V Pika Effects Crush

The following are some simple datasets/HF orgs with good datasets to test training with quickly:

• Disney Video Generation Dataset
• bigdatapw Video Dataset Collection
• Finetrainers HF Dataset Collection

Please checkout docs/models and examples/training to learn more about supported models for training & example reproducible training launch scripts.

[!IMPORTANT] It is recommended to use Pytorch 2.5.1 or above for training. Previous versions can lead to completely black videos, OOM errors, or other issues and are not tested. For fully reproducible training, please use the same environment as mentioned in environment.md.

Features

• DDP, FSDP-2 & HSDP support for all models with low-rank and full-rank training
• Memory-efficient single-GPU training
• Auto-detection of commonly used dataset formats
• Combined image/video datasets, multiple chainable local/remote datasets, multi-resolution bucketing & more
• Memory-efficient precomputation support with/without on-the-fly precomputation for large scale datasets
• Standardized model specification format for training arbitrary models
• Fake FP8 training (QAT upcoming!)

News

• 🔥 2025-03-07: CogView4 support added!
• 🔥 2025-03-03: Wan T2V support added!
• 🔥 2025-03-03: We have shipped a complete refactor to support multi-backend distributed training, better precomputation handling for big datasets, model specification format (externally usable for training custom models), FSDP & more.
• 🔥 2025-02-12: We have shipped a set of tooling to curate small and high-quality video datasets for fine-tuning. See video-dataset-scripts documentation page for details!
• 🔥 2025-02-12: Check out eisneim/ltx_lora_training_i2v_t2v! It builds off of finetrainers to support image to video training for LTX-Video and STG guidance for inference.
• 🔥 2025-01-15: Support for naive FP8 weight-casting training added! This allows training HunyuanVideo in under 24 GB upto specific resolutions.
• 🔥 2025-01-13: Support for T2V full-finetuning added! Thanks to @ArEnSc for taking up the initiative!
• 🔥 2025-01-03: Support for T2V LoRA finetuning of CogVideoX added!
• 🔥 2024-12-20: Support for T2V LoRA finetuning of Hunyuan Video added! We would like to thank @SHYuanBest for his work on a training script here.
• 🔥 2024-12-18: Support for T2V LoRA finetuning of LTX Video added!

Support Matrix

[!NOTE] The following numbers were obtained from the release branch. The main branch is unstable at the moment and may use higher memory.

Model Name	Tasks	Min. LoRA VRAM*	Min. Full Finetuning VRAM^
LTX-Video	Text-to-Video	5 GB	21 GB
HunyuanVideo	Text-to-Video	32 GB	OOM
CogVideoX-5b	Text-to-Video	18 GB	53 GB
Wan	Text-to-Video	TODO	TODO
CogView4	Text-to-Image	TODO	TODO

_{^*Noted for training-only, no validation, at resolution 49x512x768, rank 128, with pre-computation, using FP8 weights & gradient checkpointing. Pre-computation of conditions and latents may require higher limits (but typically under 16 GB).
^{^}Noted for training-only, no validation, at resolution 49x512x768, with pre-computation, using BF16 weights & gradient checkpointing.}

If you would like to use a custom dataset, refer to the dataset preparation guide here.

Featured Projects 🔥

Checkout some amazing projects citing finetrainers:

• Diffusion as Shader
• SkyworkAI's SkyReels-A1
• eisneim's LTX Image-to-Video
• wileewang's TransPixar
• Feizc's Video-In-Context

Checkout the following UIs built for finetrainers:

• jbilcke's VideoModelStudio
• neph1's finetrainers-ui

Acknowledgements

• finetrainers builds on top of & takes inspiration from great open-source libraries - transformers, accelerate, torchtune, torchtitan, peft, diffusers, bitsandbytes, torchao and deepspeed - to name a few.
• Some of the design choices of finetrainers were inspired by SimpleTuner.