Microsoft Open Sources Phi-1.5: Language Model with 1.3 Billion Parameters Surpasses LLaMa 2

2023-09-13

Microsoft Research team has achieved another breakthrough. After surpassing Meta's LLaMa with the phi-1 version in July, researchers have now released phi-1.5, an advanced language model with 1.3 billion parameters that outperforms LLaMa 2's 7 billion parameter model in multiple benchmark tests. Microsoft has decided to open-source this model.

The phi-1.5 model, with an astonishing 1.3 billion parameters, has been meticulously designed and demonstrates outstanding performance in various domains, making it the preferred choice for widespread applications. It excels in handling query formats for question answering, chat interactions, and code-related tasks.

How far can a model with a billion parameters go? It seems, very far!!!

Today, we are releasing phi-1.5, an LLM with 1.3 billion parameters, exhibiting behavior very close to larger LLMs.

While phi-1 was trained on high-quality instructional data, phi-1.5 was trained solely on synthetic data. This sets phi-1.5 apart, as its training encompasses various data sources. The learning process of this model incorporates rich data from different sources, including Python code snippets from StackOverflow, code from competitive programming contests, synthetic Python textbooks, and exercises generated by the powerful gpt-3.5-turbo-0301.

Key details of the phi-1.5 model:

  • Architecture: Transformer-based model focusing on predicting the next word.
  • Dataset size: Trained on a massive corpus containing 30 billion tokens.
  • Training tokens: The model hones its skills through training on 150 billion tokens.
  • Precision: Utilizes the fp16 precision standard.
  • GPU: Harnesses the powerful computing capabilities of 32 A100-40G GPUs.
  • Training time: Achieves its exceptional performance through 8 days of intensive training.

The Microsoft Research team claims that phi-1.5 achieves performance levels almost at the cutting edge within models of less than 10 billion parameters. Rigorous benchmark tests on common sense, language understanding, and logical reasoning position phi-1.5 as a formidable competitor.

It is worth noting that phi-1.5 surpasses Meta's LLaMa-2 7b in AGIEval scores and approaches the level of LLaMa-2 7b in the LM-Eval Harness-measured GPT4ALL benchmark suite.

RECENT AI NEWS

RECENT AI TOOLS