OpenAI releases DALL-E 3: The latest AI image synthesis model fully integrated with ChatGPT

OpenAI announced the release of DALL-E 3 on Wednesday, the latest version of its AI image synthesis model with comprehensive capabilities similar to ChatGPT. DALL-E 3 generates images by closely following and processing text within the image, such as labels and signs, which posed a challenge for previous models. Currently in research preview, DALL-E 3 will be available to ChatGPT Plus and Enterprise customers in early October.

Similar to its predecessor, DALL-E 3 is a text-based image generator that creates novel images based on written descriptions called prompts. Although OpenAI has not disclosed technical details about DALL-E 3, the core AI model of previous DALL-E versions was trained on millions of images created by human artists and photographers, some of which were sourced from stock websites like Shutterstock. DALL-E 3 may follow a similar approach but with new training techniques and more computational training time.

Based on the samples provided by OpenAI in its promotional blog, DALL-E 3 appears to outperform any other model in terms of image synthesis based on prompts. While OpenAI's examples are carefully selected, they faithfully adhere to the instructions given by the prompts, rendering objects realistically with minimal distortion. Compared to DALL-E 2, OpenAI claims that DALL-E 3 more effectively refines details, such as hands, and can create captivating images by default without resorting to "hacks" or optimized prompts.

In contrast, competitors like Midjourney excel in rendering realistic details but require significant adjustments to prompts to control the image output.

DALL-E 3 also seems to surpass its predecessor in its ability to handle text within images (a capability that other competing models like Stable Diffusion XL and DeepFloyd are improving upon). For example, a prompt containing the description "an avocado sitting on a therapist's chair, saying 'I feel empty inside,' with a nuclear-like hole in the middle" would generate a cartoonish avocado with a dialogue bubble perfectly presenting the character's lines.

It is worth noting that OpenAI states DALL-E 3 has been "natively built" on ChatGPT, serving as a feature for ChatGPT Plus to enhance image conversation as a brainstorming partner. This also means that ChatGPT will be able to generate images based on the context of the current conversation, potentially introducing new capabilities. Microsoft's Bing Chat AI assistant, which also utilizes technology from OpenAI, is expected to generate images in conversations as early as March 2022.

The original version of DALL-E was introduced in January 2021, and OpenAI released a more powerful sequel in April 2022, ushering in a groundbreaking new era for AI-generated images that garnered widespread attention from initial closed beta testers. The DALL-E model uses a technique called latent diffusion to transform noise into images "recognized" based on training from datasets and guided prompts. The same technique also enabled the release of open-weight models like Stable Diffusion in August last year.

Since DALL-E learns image concepts through training on a large dataset of human-created artworks, AI image generation technology has been controversial since last year. The technology has sparked protests from artists who fear it will replace them or replicate their styles in unethical ways. It has also led to lawsuits regarding the unauthorized use of scraped images from copyright holders as training data, as well as new copyright regulations from the US Copyright Office and US District Courts.

To address these controversies, OpenAI states that DALL-E 3 is designed to refuse generating images primarily in the style of living artists. OpenAI also provides a form for creators to opt out and prevent their images from being used for training future models. These measures seem unlikely to satisfy artists, who typically argue that AI training should be an opt-in choice rather than default inclusion in image datasets.

Currently, under US copyright policy, purely AI-generated artworks are not eligible for copyright protection, so technically, any image created using DALL-E 3 belongs to the public domain. Although OpenAI does not explicitly acknowledge this, they state, "You own the images you generate with DALL-E 3, and you don't need our permission to reproduce, sell, or otherwise use them." This marks a significant change from OpenAI's previous practice of restricting the use of DALL-E 2 images based on licensing, where they claimed "all generated works belong to OpenAI."

In terms of safety, OpenAI states that, like DALL-E 2, DALL-E 3 implements keyword and image detection filters to limit its ability to generate violent, pornographic, or hateful content. The system also rejects requests to generate images based on the names of public figures programmatically, addressing the issue that arose when the AI image generator Midjourney produced a false image of Donald Trump being arrested.

OpenAI states that they have collaborated with expert "red teams" to identify and mitigate potential risks, such as harmful biases or the creation of propaganda and misinformation. OpenAI does not mention the potential for the tool to distort historical records, although they state they are attempting to use a "trace classifier" tool to help determine if an image was generated by DALL-E 3.

Currently, we have not had access to test DALL-E 3, but OpenAI states that this AI image generator is undergoing closed beta testing. They plan to offer it to ChatGPT Plus and Enterprise customers via an API in October and release it in labs later this year.