NExT-GPT: A Large Multimodal Language Model for Text, Image, Video, and Audio

NExT-GPT, an "any-to-any" multimodal large language model (LLM) developed by the NExT Research Center at the National University of Singapore (NUS), has recently been open-sourced. This model is capable of processing text, images, videos, and audio as input or output. NExT-GPT is built upon existing pre-trained models and only requires updating 1% of the total parameters during the training process. NExT-GPT features a chat-based interface where users can input text or upload image, video, or audio files. The model can understand the input content and answer related questions or generate text, images, videos, or audio based on user requests. The system is built upon open-source pre-trained encoders and decoders, including Vicuna and Stable Diffusion, with trainable neural network layers in between. These layers are trained using a technique called MosIT (Modality Switching Instruction Training), invented by the NExT team. According to the NExT team: "In general, our research demonstrates the promising possibility of building AI agents that can model universal modalities, opening up avenues for more human-like AI research in the community." Multimodal artificial intelligence based on LLMs is an active research field. In 2022, InfoQ reported on DeepMind's Flamingo, which combines pre-trained visual and language models to answer questions about input images and videos. Earlier, OpenAI's GPT-4 was capable of processing image inputs. Additionally, Microsoft has developed two visual language models: Visual ChatGPT, which utilizes ChatGPT to invoke different visual base models for performing tasks, and LLaVA, which combines CLIP for vision and LLaMA for language and connects them through additional network layers for end-to-end visual instruction training. The architecture of NExT-GPT consists of three levels: an encoding stage with linear projection to map various input modalities to a unified space, the Vicuna LLM "core" that generates tokens including signals indicating which output modality to use, and a decoding stage composed of modality-specific transformer layers and decoders. Since the encoders, decoders, and Vicuna model are fixed, only approximately 1% of the total model parameters need to be updated during training, resulting in cost savings. Similar to Microsoft's LLaVA, NExT-GPT is trained using instruction switching. For NExT-GPT, the team at NUS created a sample dataset of conversations between human users and chatbots. The dialogues consist of three to seven exchanges and cover multiple modalities in both input and output. The team used ChatGPT and other generative AI models, including Midjourney, to assist in generating these examples. Overall, the dataset contains approximately 5,000 dialogues. The performance of NExT-GPT was evaluated on several multimodal generation benchmarks. While it did not achieve state-of-the-art results, the model performed comparably to baseline models. The team also enlisted human reviewers to score the model's outputs in various scenarios on a scale of 1 to 10. Reviewers gave higher scores in image generation scenarios compared to video and audio. Note: The HTML tags in the original text have been retained, while the style and class attributes have been removed.