AI Revolution on Mobile Devices: SnapFusion Leads the New Era

2023-06-26

Diffusion Models: The Next Breakthrough in AI If you've been closely following advancements in the field of artificial intelligence, then you've probably heard of diffusion models. They are the key to the revolution in generative AI methods. We now have models that can generate realistic images from text prompts in a matter of seconds. They have completely transformed content generation, image editing, super-resolution, video synthesis, and 3D asset generation. However, such exceptional performance comes at a cost. Diffusion models have very demanding computational requirements. This means that you need high-end graphics processors to fully leverage their capabilities. Yes, there have been attempts to make them run on local computers, but even then, you still need a high-end machine. On the other hand, using cloud service providers may be an alternative, but in such cases, you may face privacy risks. Furthermore, we need to consider the issue of mobile devices. For most people, they spend more time on their phones than on computers. If you want to use diffusion models on mobile devices, good luck, because the hardware limitations of the devices make it too demanding. Diffusion models are the next major breakthrough, but before applying them to practical applications, we need to address their complexity. There have been numerous attempts to accelerate the inference process on mobile devices, but they have not achieved a seamless user experience or quantitative evaluation of the generated quality. Until now, this has been a story because we have a new member in this field, and its name is SnapFusion. SnapFusion is the first text-to-image diffusion model that generates images on mobile devices in less than 2 seconds. It optimizes the UNet architecture and reduces the number of denoising steps to improve inference speed. Additionally, SnapFusion adopts an evolving training framework, introducing a data distillation process and enhancing learning objectives during the distillation steps. Before making any changes to the structure of SnapFusion, the redundancy of the SD-v1.5 architecture was studied to achieve an efficient neural network. However, applying traditional pruning or architecture search techniques to SD is challenging due to the high training cost. Any changes to the architecture could lead to performance degradation, requiring extensive fine-tuning with significant computational resources. Therefore, an alternative solution had to be developed to maintain the performance of the pre-trained UNet model while gradually improving its effectiveness. To increase inference speed, SnapFusion focuses on optimizing the UNet architecture, which is the bottleneck in conditional diffusion models. Existing work mainly focuses on post-training optimization, but SnapFusion identifies architectural redundancy and proposes an evolving training framework that significantly improves speed while outperforming the original Stable Diffusion model. It also introduces a data distillation process to compress and accelerate the image decoder. SnapFusion includes a robust training phase where each cross-attention and ResNet block is probabilistically executed through random forward passes. This robust training augmentation ensures network tolerance to architectural permutations, enabling accurate evaluation of each block and achieving stable architectural evolution. By training the image decoder obtained through channel reduction using synthetic data, SnapFusion achieves an efficient image decoder through the distillation process. This compressed decoder significantly reduces the number of parameters and is faster than the SD-v1.5 decoder. The distillation process involves generating two images, one from the efficient decoder and another from the SD-v1.5, obtaining latent representations from the UNet of SD-v1.5 using text prompts. The proposed progressive distillation method includes a regular distillation loss objective that minimizes the gap between student UNet predictions and teacher UNet noisy latent representations. Additionally, a CFG-aware distillation loss objective is introduced to improve CLIP scores. CFG-guided predictions are used in both teacher and student models, where CFG scale is randomly sampled, providing a trade-off between FID and CLIP scores during the training process. Thanks to the improved progressive distillation and network architecture development, SnapFusion can generate 512x512 pixel images from text prompts on mobile devices in less than 2 seconds. The generated images exhibit quality similar to state-of-the-art Stable Diffusion models. (Source: https://www.marktechpost.com/2023/06/24/the-artist-pal-in-your-pocket-snapfusion-is-an-ai-approach-that-brings-the-power-of-diffusion-models-to-mobile-devices/)

RECENT AI NEWS

RECENT AI TOOLS