New Supporters for Open Source AI

2023-06-02

When Meta's LLaMA code leaked on GitHub, researchers worldwide gained access to the first GPT-level large language model (LLM). Subsequently, a series of LLMs were launched, injecting a new dimension into open-source artificial intelligence. LLaMA provided a stage for models like Alpaca and Vicuna-13B from Stanford University, becoming a representative of open-source technology. However, a new competitor seems to have emerged - Falcon. Developed by the Abu Dhabi Technology Innovation Institute (TII) in the United Arab Emirates, Falcon outperforms LLaMA in terms of performance. It has three different variants: 1B, 7B, and 40B. According to the institute, FalconLM is the most powerful open-source language model to date. The largest variant, Falcon 40B, has 40 billion parameters, which is smaller compared to LLaMA's 6.5 billion parameters. Faisal Al Bannai, Secretary-General of the Advanced Technology Research Council (ATRC), believes that Falcon's release will break the access restrictions of LLMs, enabling researchers and entrepreneurs to provide the most innovative use cases. Top of the leaderboard Two variants of FalconLM, Falcon 40B Instruct and Falcon 40B, rank first on the Hugging Face OpenLLM leaderboard, while Meta's LLaMA ranks third. Hugging Face evaluated the performance of these models based on four popular benchmark tests: AI2 Reasoning Challenge, HellaSwag, MMLU, and TruthfulQA. Although the paper on LLM has not been published, we have learned that Falcon 40B underwent extensive training on a massive dataset, including one trillion tokens from a carefully curated refined-web dataset, ensuring its quality and relevance. Researchers revealed a particular focus on achieving high-quality data in large-scale scenarios. It is well known that LLMs are highly sensitive to the quality of their training data, so considerable effort was put into building a data pipeline capable of efficient processing across tens of thousands of CPU cores. The pipeline aims to extract top-notch content from the web, including extensive filtering and deduplication techniques. TII released the refined-web dataset, which has been proven to be highly effective. Models trained solely on this dataset can match or even surpass models trained on meticulously curated corpora, showcasing their exceptional quality and influence. The Falcon model also possesses multilingual capabilities. It can understand English, German, Spanish, and French, with limited abilities in some European languages such as Dutch, Italian, Romanian, Portuguese, Czech, Polish, and Swedish. Additionally, Falcon-40B is the second true open-source model to be released after H2O.ai. However, evaluating these two models has become challenging as H2O.ai has not undergone benchmark testing with other models on this leaderboard. Commercial applications Although the code for LLaMA is available on GitHub, its weights have never been publicly released, limiting its commercial applications. Furthermore, all variants depend on the original LLAMA license, making them unsuitable for small-scale commercial use. In contrast, Falcon uses a modified Apache license, which means these models can be fine-tuned and used for commercial purposes. Falcon becomes the first open-source large language model that surpasses research restrictions. Initially, the license stipulated a default royalty of 10% on attributed revenue exceeding one million dollars. However, it was later announced that Falcon has transitioned to the Apache 2.0 license, eliminating the need for royalty obligations. Compared to GPT-3, Falcon has achieved significant performance improvements, utilizing only 75% of the training compute budget and requiring only one-fifth of the computation during inference. Additionally, the model leverages 75% of GPT-3's training compute, 40% of Chinchilla's compute, and 80% of PaLM-62B's compute, achieving efficient utilization of computational resources. Open-source vs. closed-source While GPT-4 is the most advanced LLM to date, it is closed-source, and OpenAI has not disclosed any details about the model's architecture, size, hardware, training compute, dataset construction, training methods, etc. This prevents researchers and developers from understanding the technical details and internal workings of the model. In contrast, open-source AI promotes collaboration, transparency, and greater innovation in the field. Open-source models allow for greater collaboration and knowledge sharing, leading to faster progress and innovation, as seen since the launch of LLaMA. The availability of LLaMA enables researchers and developers to access powerful language models without investing in proprietary solutions or expensive cloud resources. Some experts criticize closed-source models for their lack of transparency and potential biases, making open-source models an alternative to closed-source models.

RECENT AI NEWS

RECENT AI TOOLS