Why Meta Should Hurry to Launch LlaMa 3

If Meta wants to stay in sync with its competitors like Falcon, now is the time for Meta to accelerate the release of LlaMa 3.

As the reigning champion in the open-source domain, Meta is going through a challenging period. The recently released Llama and Llama 2 have received acclaim for being open-source language models, but they have also led to the departure of some scientists and engineers who were previously involved in Llama development.

The reason for their departure is the internal competition for computing resources with another Meta research team developing competitive models.

While this tech giant is working hard to address internal issues, it is facing fierce competition from other companies willing to contribute to open-source projects.

A new champion has emerged in the open-source language model arena, and this champion comes from the Middle East. Falcon, recently released by TII, ranks high on the leaderboard.

Falcon 180b boasts 180 billion parameters and has been trained on a massive dataset of 35 trillion tokens, which has caught the community's attention. In terms of performance, Falcon 180b has established its position and dominates the leaderboard for open-access models. Although it is challenging to establish a definitive ranking at this early stage, Falcon 180b's performance has already sparked comparisons with PaLM-2, demonstrating its capabilities.

Only two companies left

However, Meta is not competing with anyone other than OpenAI, which is discussing multimodal capabilities and hopes to integrate its image generation model, DALLE 3.

In such an environment, discussions about LLaMa 3 are filled with various expectations and predictions. Many anticipate that LLaMa 3 will utilize high-quality training data, such as Phi 1.5, to enhance performance. People are also excited about the potential to use more tokens and further explore the scaling laws. Additionally, there are discussions about hybrid architectures, which are statistical architecture approaches that can improve performance beyond single expert or submodel systems.

People also expect LLaMa 3 to bring multimodal capabilities to the open-source community. Meta can leverage its own multimodal model ecosystem built on LLaMa, such as mPLUG-Owl, llava, minigpt4, and blip2 based on LLaMa.

Open-source projects pin their hopes on LLaMa

For small projects that rely on open-source language models (LLMs), Meta has become a key player. There are several models on the open-source language model leaderboard that are fine-tuned based on LLaMa, at least six of them, including Uni-TianYan, FashionGPT, sheep-duck, Orca, and GenZ Model developed by Indian developers.

Although Falcon provides a very good and powerful alternative, there are concerns about its license.

Essentially, the terms imply that the licensor reserves the right to modify the "acceptable use policy" without explicit notice to users, and users must comply with the latest version of the policy. Failure to do so may result in a violation of the license terms.

From Reddit to hacker news forums, several forums recognize Meta's significant role in open-source language model development and express disappointment over the delay of LLaMa 3. According to an article in The Wall Street Journal, Meta has not even started training it yet and will launch the project in early to mid-2024.

This delay also means that the open-source community will lag behind. Commentators reiterate that Meta's actions have a significant impact on the availability of this model. If Meta chooses not to release LLaMa 3 as open-source, other experienced and well-funded teams are unlikely to abandon models developed at a cost of millions of dollars, except for Falcon.