Cerebras Systems, Petuum, and MBZUAI Collaborate to Launch Open Source LLM360

AI supercomputer company Cerebras Systems, AI company Petuum, and Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) have launched LLM360, a framework for creating open-source large language models (LLM). LLM360 was developed in collaboration with MBZUAI's Institute for Foundational Model Research, and it promises to simplify, accelerate, and reduce the cost of LLM development by providing detailed insights and methodologies to developers.

Two open-source large language models have been released: Amber, an English model trained on 1.2 trillion tokens with 7 billion parameters, and CrystalCoder, another 7 billion parameter model trained on 1.4 trillion tokens, designed specifically for English language and coding tasks. Both models are released under the Apache 2.0 license. There is also a model called Diamond with 650 billion parameters, which will be released soon. These models were trained on the Condor Galaxy 1 supercomputer built by G42 and Cerebras Systems.

Both models are built on Meta's LLaMA architecture and Amber is said to perform similarly to LLaMA-7B and OpenLLaMA-v2-7B in terms of performance, and outperforms Pythia-6.7B.

CrystalCoder has undergone careful training, incorporating a thoughtful blend of text and code data to enhance its effectiveness in both domains. It is worth noting that the introduction of code data occurred early in the pre-training phase, unlike Code Llama 2, which relies solely on code data for fine-tuning on Llama 2. Additionally, CrystalCoder is specifically trained for Python and web programming languages, with the aim of enhancing its capabilities as a programming assistant.

UAE is moving towards AI dominance

With recent AI developments, UAE is striving to become an AI superpower. Following TII's Falcon and the Jais large language models targeting specific populations, UAE has also been promoting open-source models to foster research initiatives. With the recent launch of AI company AI71, UAE even appears to be competing with AI giant OpenAI.