Berkeley Open Source AI Image Editing Model InstructPix2Pix

2023-07-19

Researchers from the Berkeley Artificial Intelligence Research (BAIR) Lab have open-sourced InstructPix2Pix, a deep learning model that can edit images based on human instructions. InstructPix2Pix was trained on synthetic data and outperforms baseline AI image editing models. The BAIR team presented their work at the recent IEEE/CVF Computer Vision and Pattern Recognition (CVPR) 2023 conference. They first generated a synthetic training dataset consisting of paired images and editing instructions to transform the first image into the second image. This dataset was used to train an image generation diffusion model. The result is a model that, given a source image, can accept text-based instructions on how to edit the image. For example, given an image of a person riding a horse and the prompt "make her ride a dragon," it will output the original image with the horse replaced by a dragon. According to the BAIR researchers, "despite being trained entirely on synthetic examples, our model achieves zero-shot generalization to arbitrary real images and natural human-written instructions. Our model supports intuitive image editing and can perform various sets of edits based on human instructions, such as object replacement, style change, scene manipulation, artistic media, etc." Early AI work for image editing was often based on style transfer, and popular text-to-image generation models like DALL-E and Stable Diffusion also support image-to-image style transfer operations. However, performing targeted edits using these models has been challenging. Recently, InfoQ introduced Microsoft's Visual ChatGPT, which can invoke external tools to edit images based on textual descriptions. To train InstructPix2Pix, BAIR first created a synthetic dataset. For this, the team fine-tuned GPT-3 on a small dataset composed of human-written examples, consisting of input captions, editing instructions, and desired output captions. Then, this fine-tuned model was given a large dataset of input image captions, from which over 450k editing and output captions were generated. The input and output captions were then provided to a pre-trained prompt-to-prompt model, which generated paired similar images based on the captions. Based on this dataset, the researchers trained InstructPix2Pix based on Stable Diffusion. To evaluate its performance, the team compared its output with the baseline model SDEdit. They balanced between two metrics: consistency, which is the cosine similarity between the CLIP embeddings of the input image and the edited image, and direction similarity, or the consistency between the changes in the editing captions and the changes in the edited image. In the experiments, for a given direction similarity value, InstructPix2Pix produced more consistent images compared to SDEdit. In his deep learning newsletter "The Batch," AI researcher Andrew Ng commented on InstructPix2Pix: "This work simplifies modifying generated and synthetic images and provides more coherent results during editing. Cleverly leveraging pre-existing models allowed the authors to train their model on new tasks with relatively fewer manually labeled examples." The code for InstructPix2Pix can be found on GitHub. The model and web-based demo are available on Huggingface. (Source: https://www.infoq.com/news/2023/07/berkeley-instruct-pix2pix/)

RECENT AI NEWS

RECENT AI TOOLS