Adept releases Fuyu-8B, a new model for multimodal AI agents

Adept has released Fuyu-8B, a streamlined version of its multimodal model, now available through HuggingFace. This model is capable of understanding charts, documents, and graphs, with improved OCR functionality. The new model has garnered significant attention for several key reasons, including its simplified architecture. Compared to other multimodal models, Fuyu-8B offers a simpler training process, providing users with a more accessible, scalable, and deployable solution. Specifically designed for digital AI agents, this model caters to the specific needs of digital agents. It can handle images of any resolution, answer queries related to graphics, charts, and user interface-based questions, and can perform precise localization within screen images. Notably, Fuyu-8B demonstrates impressive speed, providing responses for large images in less than 100 milliseconds. Despite being optimized for specific applications, Fuyu-8B performs exceptionally well in standard image understanding benchmarks, including visual question answering and natural image captioning tasks. The Fuyu model avoids complex architectural designs found in other models and instead adopts a Transformer structure consisting only of decoders, eliminating the need for a separate image encoder. Image patches are linearly projected into the first layer of the Transformer model, simplifying the model's structure. This simplified architectural design allows Fuyu to support images of any size, treating image tokens the same way as text tokens. Special image line break tokens indicate line breaks, while the model utilizes existing positional embeddings to adapt to different image sizes. This approach eliminates the need for separate high-resolution and low-resolution training stages, greatly simplifying the training and inference processes. To evaluate these improvements, Adept conducted evaluations on well-known image understanding datasets, including VQAv2, OKVQA, COCO Captions, and AI2D. Fuyu-8B demonstrated strong performance, even in the domain of natural images. Despite having significantly fewer parameters, it outperformed models such as QWEN-VL and PALM-e-12B in multiple metrics. Even the Fuyu-Medium variant proved comparable to PALM-E-562B. While PALI-X remains ahead in these benchmarks due to fine-tuning for each specific task, it is important to note that Adept's primary focus is not on optimizing these benchmarks. However, Fuyu-8B and its variants offer a simpler yet efficient alternative as emerging choices in the field of multimodal models.