Can Stability AI and Meta Meet the Multimodal Challenge of OpenAI?

OpenAI has just revealed that ChatGPT can now see, speak, and listen, making it a truly multimodal system. In addition, they also plan to add DALL.E-3 to ChatGPT and ChatGPT Enterprise. Meanwhile, Google is also doing something similar with their own multimodal system Gemini, which will be launched this fall.

It is expected that there will be two multimodal products by October, and it will be interesting to see the contributions of open-source participants in the multimodal market. Currently, Stability AI and Meta seem to be strong competitors that can achieve this goal.

Stability AI's Conditions

Stability AI has the necessary resources to create an open-source multimodal model. They have Stable Diffusion for text-to-image, Stable LM for text-to-text, and the recently added Stable Audio for text-to-music generation. By combining these three models, Stability AI may create a multimodal model similar to OpenAI's. Although Stable Audio is not open-source, Stability AI has revealed plans to launch an open-source model based on the Stable Audio architecture, which uses different training data.

In addition, earlier this year, Stability AI and its multimodal AI research lab DeepFloyd announced the research release of DeepFlyd IF, a powerful text-to-image cascaded pixel diffusion model. It would not be surprising to see Stability AI's multimodal offering soon.

Meta's Plans

Surprisingly, recently during a social event, OpenAI engineer Jason Wei overheard a conversation that hinted at Meta having accumulated enough computing power to train Llama 3 and Llama 4. While the goal of Llama 3 is to achieve performance comparable to GPT-4, it will remain free. Additionally, it is expected that Llama 3 will introduce open-source multimodal capabilities.

ImageBind is part of Meta's plan to create a multimodal artificial intelligence system that can learn from all possible types of data in its surroundings. ImageBind is the first AI model that can bind information from six modalities. The model can learn single embeddings or shared representation spaces, not only for text, images, videos, and audio, but also for depth (3D) sensors, as well as heat (infrared radiation) and inertial measurement units (IMU) used for computing motion and position.

In addition, Meta has released a multimodal model called "CM3leon" that can generate both text-to-image and image-to-text. Furthermore, Meta's seamless M4T can perform speech-to-text, speech-to-speech, text-to-text translation, and speech recognition tasks in up to 100 languages.

Multimodal is the Future

Open-source LLMs like LLaVA can be customized to meet the specific needs of organizations, helping to reduce the cost of developing and maintaining AI applications. The lack of true multimodal options in the open-source market has led developers to experiment with multimodality on their own. Some attempts have been successful, while others have not. However, this trial-and-error approach is inherent to the open-source community.

Earlier this year, a group of scientists from the University of Wisconsin-Madison, Microsoft Research, and Columbia University created a multimodal called LLaVA. It is a multimodal LLM that can handle text and image inputs. It uses Vicuna as the large language model (LLM) and CLIP-ViT-L/14 as the visual encoder.

Another group of researchers from Abdullah King University has created MiniGPT-4, an open-source model that performs complex visual language tasks like GPT-4. To build MiniGPT-4, the researchers used Vicuna based on LLaMA as the language decoder and BLIP-2 visual language model as the visual decoder. Additionally, to simplify the process of creating multimodal models, the open-source community has introduced models like BLIP-2 and mPLUG-Owl.

While the open-source community is experimenting to create a viable multimodal system, Meta and Stability AI still need to accelerate their pace and develop multimodal solutions as soon as possible. Otherwise, Google and OpenAI may take the lead, further widening the gap between open-source and closed-source participants.