Researchers have used GPT-4 to generate an instruction tracking dataset that includes a virtual dialogue between a human user and an AI assistant regarding image content. This dataset was used to fine-tune the LLaVA model, which consists of two base models, CLIP (visual) and LLaMA (language), and an additional network layer that connects them. The team also used GPT-4 to evaluate LLaVA's response in the experiment by asking it to rate LLaVA's output on a scale of 1 to 10. After further fine-tuning on the Science QA training dataset, LLaVA achieved a record-breaking accuracy of 92.53%.
The technique of fine-tuning large language models (LLMs) using instruction tracking datasets has been shown to improve performance, as demonstrated by ChatGPT, and has prompted researchers to explore the application of this technique in smaller LLMs. InfoQ recently reported on LLaMA, which, with only 7B parameters compared to GPT-3's 175B, outperforms GPT-3 in many tasks. The next step in the development of AI assistants is to enhance their ability to process image data, as demonstrated by the release of GPT-4 and Visual ChatGPT.
The goal of the LLaVA team is to train models end-to-end through visual instruction fine-tuning. To achieve this, researchers extracted images from the COCO dataset. With the images annotated with titles and object bounding boxes, the team inputted this data into the text-only GPT-4 along with prompts, requesting GPT-4 to output instruction tracking data, including virtual dialogues between a human user and an AI assistant regarding image content, questions about image content details, and questions requiring inference of image content. Overall, the generated dataset consists of 158K samples.
Researchers from Microsoft, the University of Wisconsin-Madison, and Columbia University have open-sourced the Large Language and Vision Assistant (LLaVA). LLaVA is based on the CLIP image encoder and LLaMA language decoder, fine-tuned on a synthetic instruction-following dataset, and achieves state-of-the-art accuracy on the Science QA benchmark test.
The LLaVA architecture consists of a CLIP base model and a projection matrix layer that converts images into word embedding space; text input is also transformed into the same space. The image and word tokens are then passed to the LLaMA decoder, which generates the output. The pre-training process first trains the projection matrix, and then the fine-tuning process updates the projection layer and LLaMA decoder weights, while the CLIP weights are frozen.
Chunyuan Li, the author of LLaVA, has answered some questions about this work on Twitter. When some users compared LLaMA to MiniGPT-4, Chunyuan Li pointed out that LLaMA is able to reproduce the image-based results of the GPT-4 paper, while MiniGPT-4 cannot. He also stated, "LLaVA has rigorous quantitative results, including similarity to Visual Chat and GPT-4, state-of-the-art accuracy on Science QA, and ablation studies on data iteration and model design. On the other hand, Mini GPT-4 lacks quantitative results... Finally, it should be noted that the focus of this line of work is on data center, not model center. As the differences between models become smaller, the impact of data quality on results becomes greater. We have made our multimodal instruction-following data publicly available to replicate multimodal GPT-4."
The source code for LLaVA is available on GitHub, and an interactive demo is provided on the project website. The training data and model weights for LLaVA can be obtained from Huggingface. The model uses incremental weights on top of LLaMA and "should not be used for purposes other than research."
Source: https://www.infoq.com/news/2023/05/microsoft-llava-chatbot/?topicPageSponsorship=0d3ead73-1914-40fe-bf33-ff2dbeb0a5be&itm_source=presentations_about_ai-ml-data-eng&itm_medium=link&itm_campaign=ai-ml-data-eng
Alphabet Q1 paid subscribers reached 350M, fueled by YouTube & Google One. YouTube ad revenue slightly missed at $9.88B amid Premium growth. Total Q1 revenue hit $109.9B, beating forecasts.
Anthropic releases Claude connectors for Adobe, Blender, and Ableton, enabling AI workflow automation and app control. It also sponsors the Blender Development Foundation.
Apple introduces an App Store model splitting 12-month plans into monthly installments. On iOS 26.4+, it lowers upfront costs without reducing total price, adding billing tracking.
Google launches Aadhaar digital IDs in Wallet for India, enabling secure QR/NFC verification. A phased partner rollout targets key sectors, while passport-based credentials expand globally.
YouTube tests "Ask YouTube," an AI tool blending text and video for complex queries. Initially for US Premium users, it supports interactive follow-ups, expanding Google's AI search.
Google grants US DoD classified AI access, joining OpenAI and xAI in defense deals amid Anthropic's dispute. Contractual safeguards against surveillance and lethal weapons remain unenforceable.
Mistral AI launches Workflows, a Temporal-powered orchestration platform for secure, Python-driven enterprise AI automation featuring human-in-the-loop approvals.