Single-Step Diffusion Model Breakthrough for Rapid Image Synthesis

In the current era of artificial intelligence, computers can generate their own "art" through diffusion models, which involves adding structure to noisy initial states through iterations until clear images or videos emerge.

Diffusion models have suddenly become the focus of attention for everyone: with just a few words, one can immediately experience the dopamine surge at the intersection of reality and fantasy. However, behind the scenes, this requires a complex and time-consuming process that involves multiple iterations of algorithms to perfect the images.

Researchers at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) have introduced a new framework that simplifies the multi-step process of traditional diffusion models into a single step, overcoming previous limitations. This is achieved through a teacher-student model, where a new computer model is taught to mimic the behavior of more complex and primitive image generation models.

This approach is called Distribution Matching Distillation (DMD), which preserves the quality of generated images and allows for faster generation speed.

"Our work is a novel approach that can accelerate current diffusion models like Stable Diffusion and DALLE-3 by 30 times," said Wei-Yin Tian, a doctoral student in electrical engineering and computer science at MIT, a researcher at CSAIL, and the main researcher of the DMD framework.

"This advancement not only significantly reduces computation time, but also maintains the quality of generated visual content, and may even surpass its quality. In theory, this method combines the principles of generative adversarial networks (GANs) with diffusion models, achieving one-step generation of visual content, which is in stark contrast to the hundreds of iterations required by current diffusion models for refinement. It may become a new generation modeling method that performs excellently in terms of speed and quality."

This single-step diffusion model can enhance design tools, enabling faster content creation and potentially supporting advancements in fields such as drug discovery and 3D modeling, where speed and efficiency are crucial.

DMD cleverly incorporates two components. First, it uses regression loss, which anchors the mapping to ensure rough organization of image space, making the training more stable.

Next, it uses distribution matching loss to ensure that the probability of generating a given image using the student model corresponds to its frequency of occurrence in the real world. To achieve this, it utilizes two diffusion models as guidance, helping the system understand the differences between real and generated images, and enabling faster training of the one-step generator.

The system minimizes the distribution difference between the generated images and the training dataset used by traditional diffusion models by training a new network, allowing for faster generation. "Our key insight is to use two diffusion models to approximate the gradients for the improvement of the new model," said Wei-Yin Tian.

"In this way, we distill the knowledge of the original, more complex model into a simpler and faster model, while bypassing the notorious instability and mode collapse issues in GANs."

Wei-Yin Tian and his colleagues used a pre-trained network as the new student model, simplifying the process. By copying and fine-tuning the parameters of the original model, the team achieved fast convergence of the new model, which is capable of generating high-quality images with the same architectural foundation. "This enables the combination of the original architecture with other system optimizations, further accelerating the creation process," added Wei-Yin Tian.

When extensively benchmarked against conventional methods, DMD consistently demonstrated performance. In popular benchmark tests that generate images based on specific categories on ImageNet, DMD is the first one-step diffusion technique that can generate images almost identical to those of the original, more complex model, with an impressive Fréchet Inception Distance (FID) score of only 0.3. FID is primarily used to evaluate the quality and diversity of generated images.

In addition, DMD performs exceptionally well in industrial-scale text-to-image generation and achieves state-of-the-art single-step generation performance. There is still a slight quality gap when dealing with more challenging text-to-image applications, indicating room for further improvement in the future.

Furthermore, the performance of DMD in generating images is closely related to the capability of the teacher model used during distillation. In its current form, using Stable Diffusion v1.5 as the teacher model, the student inherits limitations such as detailed rendering of text and small faces, suggesting that more advanced teacher models can further enhance the images generated by DMD.

Fredo Durand, a professor of electrical engineering and computer science at MIT and the chief researcher at CSAIL, is the first author of this paper. "We are very excited to finally achieve one-step image generation, which will greatly reduce computational costs and speed up the process."

"Finally, a paper successfully combines the diversity and high visual quality of diffusion models with the real-time performance of GANs," said Alexei Efros, a professor of electrical engineering and computer science at the University of California, Berkeley, who was not involved in this research. "I expect this work to open up great possibilities for high-quality real-time visual editing."