Google DeepMind launches DRaFT: Direct Reward Fine-Tuning method for optimizing diffusion models

The diffusion model has brought revolutionary changes in the generation modeling of various data types. However, in some practical applications, such as generating beautiful images from text descriptions, fine-tuning is often required. Text-to-image diffusion models employ techniques such as classifier-free guidance and curated datasets (such as the LAION aesthetic dataset) to improve alignment and image quality.

In their research, the authors propose a simple and efficient gradient reward fine-tuning method that fine-tunes the diffusion model by propagating gradients throughout the sampling process. They introduce the concept of Direct Reward Fine-Tuning (DRaFT), which involves backpropagating through an unfolded computation graph of length 50 to compute gradients. To manage computations and memory efficiently, they employ gradient checkpointing and optimize LoRA weights instead of modifying the entire model parameters.

The above image illustrates the process of using a human preference reward model for DRaFT. Additionally, the authors make some improvements to the DRaFT method to enhance its efficiency and performance. Firstly, they propose DRaFT-K, an improved version that only backpropagates through the last K steps of sampling when computing fine-tuning gradients. Experimental results demonstrate that this truncated gradient method better prevents gradient explosion compared to full backpropagation in the same training steps.

Furthermore, the authors introduce DRaFT-LV, an improved version of DRaFT-1, which calculates gradients with lower variance by averaging over multiple noisy samples, further improving the efficiency of the method.

The researchers apply DRaFT to Stable Diffusion 1.4 and evaluate it using various reward functions and prompt sets. Their method exhibits significant efficiency advantages over reinforcement learning-based fine-tuning algorithms when it comes to utilizing gradients for fine-tuning. For example, compared to reinforcement learning algorithms, DRaFT achieves over 200 times speed improvement in maximizing LAION aesthetic classifier scores.

DRaFT-LV, an improved version proposed by the authors, demonstrates outstanding efficiency, with a learning speed approximately twice that of the previous gradient fine-tuning method ReFL. Additionally, the authors showcase the flexibility of DRaFT, as they can combine or interpolate the DRaFT model with pre-trained models by adjusting LoRA weights.

In conclusion, direct fine-tuning of diffusion models based on differentiable rewards provides a promising approach for improving generative modeling techniques, not only in applications such as images and text but also in broader domains. The efficiency, flexibility, and effectiveness of the DRaFT method make it an indispensable tool for researchers and practitioners in the fields of machine learning and generative modeling.