LLaVA-1.5: Open-Source Multi-Modal Large Model Challenge GPT-4V

LLaVA-1.5, a multimodal large-scale model jointly developed by researchers from the University of Wisconsin-Madison, Microsoft Research, and Columbia University, has recently demonstrated new state-of-the-art (SOTA) performance in 11 benchmark tests, covering tasks such as visual question answering and image description. This showcases its powerful multimodal understanding capabilities, making it a strong competitor to OpenAI's GPT-4V. Compared to earlier versions of LLaVA, LLaVA-1.5 incorporates higher-resolution image inputs and a larger-scale language model with 13 billion parameters. It has been trained on a visual question answering dataset tailored for academic tasks, resulting in significant performance improvements. Researchers have found that, in enhancing multimodal capabilities, fine-tuning with visual instructions plays a more crucial role compared to large-scale pretraining. Surprisingly, LLaVA-1.5 adopts a very simple architectural design, requiring only training a fully connected layer on publicly available 6 million image-text pairs. Despite its simplicity, it outperforms complex SOTA systems and sets new records in multiple benchmark tests. Training LLaVA-1.5 using 8 A100 GPUs takes only one working day. This research demonstrates that open-source large-scale models can achieve high competitiveness through simple architectural designs and the use of public data, providing valuable insights for future research. LLaVA-1.5 has attracted industry attention as a "hard competitor to GPT-4" due to its outstanding multimodal understanding capabilities. Recently, OpenAI announced the removal of multimodal capability restrictions on ChatGPT. The remarkable abilities of multimodal GPT-4V have led people to compare it with GPT-4.5. However, LLaVA-1.5 quickly emerged as an open-source competitor to GPT-4V. Researchers significantly improved the performance of LLaVA-1.5 by using CLIP-ViT-L-336px with MLP mapping and introducing simple prompt formats in a visual question answering dataset tailored for academic tasks. Experimental results show that LLaVA-1.5's design is simpler, surpassing Qwen-VL trained on 1.45 billion data and HuggingFace IDEFICS trained on 130 million data (similar to Flamingo's 80B model) using only 1.2 million publicly available data. Additionally, training the 13 billion parameter model only requires 8 A100 GPUs and can be completed within a day. In summary, LLaVA-1.5 is an open-source multimodal large-scale model that achieves SOTA performance and demonstrates powerful multimodal understanding capabilities, making it a strong competitor to GPT-4V. Through simple architectural design and the use of public data, LLaVA-1.5 breaks records in multiple benchmark tests, providing valuable insights for future research.