Diffusion Models Drive Novel Breakthroughs in Reinforcement Learning Field

The diffusion model has attracted much attention due to its excellent image generation capabilities, and has now become the main architecture in popular text-to-image models such as DALL-E, Stable Diffusion, and Midjourney. However, the application of diffusion models is not limited to image generation. Recently, researchers from Meta, Princeton University, and the University of Texas at Austin published a new paper pointing out the important value of diffusion models in the field of reinforcement learning.

This paper introduces a technique that uses diffusion-based world models (DWM) to train reinforcement learning agents. DWM effectively improves the performance of current model-based reinforcement learning systems by predicting the future states of the environment for several steps.

Reinforcement learning can be divided into two types: model-free and model-based. Model-free reinforcement learning learns policies or value functions directly from interactions with the environment, without predicting future states of the environment. On the other hand, model-based reinforcement learning trains policies by simulating the environment. The advantage of this approach is that it requires fewer real environment data samples, making it particularly suitable for applications such as autonomous driving cars and robots that require a large amount of experimental data. However, model-based reinforcement learning is highly dependent on the accuracy of the world model. If the model is inaccurate, the performance of the model-based reinforcement learning system may be worse than that of the model-free system.

Traditional world models use single-step dynamic prediction, which predicts rewards and the next state based only on the current state and action. However, this method may lead to inaccurate long-term predictions due to the accumulation of small errors during long-term planning. To solve this problem, DWM adopts a method of predicting multiple future steps at once. This method effectively reduces errors in long-term predictions and improves the performance of model-based reinforcement learning algorithms.

The working principle of diffusion models is to generate data by reversing a process of gradually adding noise. During the training process, the model gradually adds noise layers to the data and then tries to reverse this process and predict the original data. By repeating this process and adding more noise layers, the model learns to create detailed data from pure noise. Conditional diffusion models add an additional layer of control by associating the model's output with specific inputs, such as image-related captions, allowing the model to generate corresponding data.

Although diffusion models have shown excellent performance in image generation, they can also be applied to other types of data. In the field of reinforcement learning, DWM uses diffusion models to predict the states and rewards of multiple future steps. This method allows the agent to simulate the environment more accurately and formulate corresponding strategies.

The DWM framework consists of two training stages. In the first stage, the diffusion model is trained on a series of trajectories collected from the environment. Through this process, DWM learns a powerful world model that can predict multiple steps at once. This makes it more stable in long-term simulations compared to other model-based methods. In the second stage, the actor-critic algorithm and the diffusion world model are used to train the offline reinforcement learning strategy. This approach eliminates the need for online interaction during training, improves training speed, and reduces costs and risks.

To validate the effectiveness of DWM, researchers compared it with model-based and model-free reinforcement learning systems. The experimental results show that DWM improves performance by 44% compared to single-step world models. In addition, when the single-step world model is applied to model-free reinforcement learning algorithms, performance is usually reduced. However, when combined with DWM, the performance of the model-free reinforcement learning system exceeds the original version.

This research demonstrates the broader application trend of diffusion models in non-generative tasks. In the past year, significant breakthroughs have been made in robotics research due to the progress of generative models. Language models are helping to bridge the gap between natural language commands and robot motion commands. At the same time, transformers are helping researchers integrate data collected from different forms and settings, and train models that can be generalized to different robots and tasks. With the continuous development of these technologies, we can expect to see more innovative applications of reinforcement learning.