Google AI launches Imagen Editor and EditBench, enhancing image restoration technology

2023-06-16

Recently, people have become increasingly curious about tools that can convert text into images. While these generative models are practical, it is often unlikely to achieve satisfactory results on the first try, especially for clients with more refined creative or design requirements. Using text guidance for image editing can make the creative process more flexible. The challenge lies in generating modifications that are both consistent with the text description and coordinated with the original image. Google's research team has developed Imagen Editor, a layered diffusion model that can perform image inpainting based on text guidance. Imagen Editor is able to determine the parts that need to be inpainted during training by using object detectors, thus generating modifications that are consistent with the text description. Imagen Editor also captures the smallest details in the input image by combining the layered process with the original high-resolution image. To better evaluate the effectiveness, the Google research team has also provided EditBench, a standardized testing platform for text-guided image inpainting. EditBench evaluates the changes after inpainting by analyzing objects, attributes, and scenes in both real and synthetic images. In-depth manual evaluations on EditBench have shown that using object masks during training can significantly improve the alignment between text and images, with Imagen Editor outperforming DALL-E 2 and Stable Diffusion. Overall, these models have an advantage in rendering objects compared to rendering text and handling material/color/size attributes, but are slightly inferior in handling quantity/shape attributes. To modify an image, use Imagen Editor, a diffusion-based model optimized specifically for Imagen. It is dedicated to accurately expressing language input, specific instructions, and high-quality output. Image editing, binary masks for identifying editing areas, and text prompts are the three inputs used by Imagen Editor to determine the output sample. Image Editor allows users to make targeted changes to specific areas of an image based on masks and a set of instructions. The model takes into account the user's goals and makes realistic adjustments to the image. Imagen Editor is a text-guided image editor that combines extensive language expression with fine-grained control to generate high-quality results. Imagen Editor is an enhanced version of Imagen, fine-tuned for text-guided image inpainting using a cascaded diffusion model. With three convolutional downsampling image encoders, Imagen Editor provides more image and mask context for each diffusion stage. The reliable text-guided image inpainting in Imagen Editor is based on three fundamental methods: 1. Object mask strategy and object detector module are used during training to generate object masks, instead of the random boxes and brush masks used by previous inpainting models. 2. Improving high-resolution editing by requiring full-resolution, channel-wise concatenation of input images and masks during both training and inference. 3. Conditional Fused Generation (CFG) is used in the inference process to interpolate between conditional and unconditional model predictions, achieving high accuracy in text-guided image inpainting. Ensuring that the generated output is consistent with the text prompt is a major challenge in text-guided image inpainting. EditBench created a new benchmark for text-guided image inpainting using 240 photos. Each image is associated with a mask indicating the areas to be modified during the inpainting process. To help users specify the content of the modifications, three text prompts are provided for each image-mask pair. EditBench is a curated benchmark for text-to-image creation, similar to DrawBench and PartiPrompts, aiming to capture different categories and difficulty factors in image collection. EditBench includes both natural photos collected from existing computer vision datasets and synthetic images generated by text-to-image models, with equal quantities of both. EditBench supports a wide range of mask sizes, including large masks that extend to the image boundaries. The problem settings in EditBench aim to evaluate the performance of models on subtle details in three categories: 1. Attributes (such as material, color, shape, size, and quantity) 2. Object types (common, rare, and text rendering) 3. Scenes (indoor, outdoor, real, or painted) Text-image alignment and image quality on EditBench have undergone rigorous manual testing by the research team. In addition, they compared human preferences with computational metrics. They analyzed four models: - Image Editor (IM) - Imagen EditorRM (IMRM) - Stable Diffusion (SD) - DALL-E 2 (DL2) To evaluate the benefits of object masks during training, the researchers compared Imagen Editor with Imagen EditorRM. To compare our work with others and examine the limitations of the current state-of-the-art more broadly, we also evaluated Stable Diffusion and DALL-E 2. In summary, the provided image editing models are part of a larger class of generative models that offer capabilities previously unattainable in content generation. However, they also have the potential to generate content that may be harmful to individuals or society as a whole. In language modeling, text generation models are generally believed to inadvertently reflect and amplify social biases present in the training data. Imagen Editor is an improved version of Imagen for text-guided image inpainting. It relies on object mask strategies during the training process and additional convolutional layers for high-resolution editing. EditBench is a large-scale, systematic evaluation for text-guided image inpainting. EditBench comprehensively tests inpainting systems based on attributes, object-based, and scene-based repairs. Source: https://www.marktechpost.com/2023/06/14/google-ai-unveils-imagen-editor-and-editbench-to-improve-and-evaluate-text-guided-image-inpainting/

RECENT AI NEWS

RECENT AI TOOLS