Recently, Meta shared its latest research on CM3leon, a transformer-based model that achieves state-of-the-art results in text-to-image generation and demonstrates new capabilities of multimodal AI. CM3leon marks the first time an autoregressive model matches the performance of leading diffusion-based models on key benchmarks.
In recent years, generative AI models that can generate images from textual prompts have made rapid progress. Models like Midjourney, DALL-E 2, and Stable Diffusion can create realistic scenes and portraits from short text descriptions. These models use a technique called diffusion, which gradually reduces noise in an image and progressively brings it closer to the desired target. While diffusion-based methods have produced impressive results, their computational intensity poses challenges as they are expensive to run and often lack the speed required for real-time applications.
CM3leon takes a different approach. As a transformer-based model, it harnesses the power of attention mechanisms to measure the relevance of input data, whether it's text or images. This architectural difference allows CM3leon to achieve faster training speed and better parallelization, making it more efficient than traditional diffusion-based methods.
CM3leon efficiently trains on a single TPU pod using a licensed image dataset and achieves a FID score of 4.88 on the MS-COCO dataset. Meta researchers claim that the model is more than five times more efficient than ordinary transformer architectures.
However, the original performance metrics do not fully capture the essence of CM3leon's capabilities. CM3leon truly shines when handling more complex prompts and image editing tasks. For example, CM3leon can accurately render an image based on a prompt like "a small cactus wearing a straw hat and neon sunglasses in the Sahara Desert."
The model is also adept at editing existing images based on free-form textual instructions, such as changing the color of the sky or adding objects in specific locations. These abilities far surpass those of the leading DALL-E 2 model.
CM3leon's versatile architecture allows it to flexibly transition between text, images, and composite tasks. In addition to text-to-image generation, CM3leon can generate captions for images, answer questions about image content, and even create images based on textual descriptions of bounding boxes and segmentation maps. This ability to combine multiple modalities in a single model is unprecedented in publicly disclosed AI systems.
Given a textual description of image bounding box segmentation, CM3leon can generate an image.
Integrating a single super-resolution with CM3leon's output significantly improves image resolution and details. Here are four example images corresponding to the following prompts:
(1) A steaming cup of coffee with mountains in the background. Rest during a road trip.
(2) Beautiful and majestic sunset road. Aesthetic.
(3) A small circular island in the middle of a lake. Forest surrounds the lake. High contrast.
CM3leon's success can be attributed to its unique architecture and training methods. The model adopts a transformer architecture with only a decoder, similar to established text-based models, but with added capabilities to handle both text and images. Training involves retrieval augmentation, building on the latest work in the field, and fine-tuning the model on various image and text generation tasks.
By applying a technique called supervised fine-tuning across multiple modalities, Meta significantly improves CM3leon's performance in image captioning, visual QA, and text-based editing. Despite being trained on only 3 billion text tokens, CM3leon matches or surpasses the results of models trained on up to 100 billion tokens.
Meta has not announced plans for the public release of CM3leon. However, the model sets a new standard for multimodal AI and showcases the power of techniques like retrieval augmentation and supervised fine-tuning. It is an impressive achievement that points to a future where AI systems can seamlessly transition between understanding, editing, and generating images, videos, and text.
Source: https://www.maginative.com/article/meta-unveils-cm3leon-a-breakthrough-ai-model-for-advanced-text-to-image-generation-and-image-understanding/
Alphabet Q1 paid subscribers reached 350M, fueled by YouTube & Google One. YouTube ad revenue slightly missed at $9.88B amid Premium growth. Total Q1 revenue hit $109.9B, beating forecasts.
Anthropic releases Claude connectors for Adobe, Blender, and Ableton, enabling AI workflow automation and app control. It also sponsors the Blender Development Foundation.
Apple introduces an App Store model splitting 12-month plans into monthly installments. On iOS 26.4+, it lowers upfront costs without reducing total price, adding billing tracking.
Google launches Aadhaar digital IDs in Wallet for India, enabling secure QR/NFC verification. A phased partner rollout targets key sectors, while passport-based credentials expand globally.
YouTube tests "Ask YouTube," an AI tool blending text and video for complex queries. Initially for US Premium users, it supports interactive follow-ups, expanding Google's AI search.
Google grants US DoD classified AI access, joining OpenAI and xAI in defense deals amid Anthropic's dispute. Contractual safeguards against surveillance and lethal weapons remain unenforceable.
Mistral AI launches Workflows, a Temporal-powered orchestration platform for secure, Python-driven enterprise AI automation featuring human-in-the-loop approvals.