"Microsoft's Small AI Model Orca-Math Outperforms Large-Scale Models in Math Tests"

A group of AI researchers at Microsoft has recently reported that their developed small language model, Orca-Math, has performed exceptionally well in standardized math tests, surpassing other large-scale models. The team has published a paper on the academic preprint platform arXiv, providing detailed insights into Orca-Math's performance in the GSM8K benchmark test and comparing it with well-known large language models (LLMs).

Large language models, such as the popular ChatGPT, are renowned for their exceptional conversational abilities. However, what is less known is that most of them also possess the capability to solve mathematical word problems. Researchers evaluated these models' abilities in this aspect using the GSM8K test set, which consists of 8,500 elementary-level math word problems that require multi-step reasoning for solutions and provide standard answers.

In this new study, Microsoft's research team conducted tests on Orca-Math. Orca-Math is another AI application developed by a separate team at Microsoft specifically designed to solve math word problems. They compared Orca-Math's performance with larger AI models.

Microsoft highlighted in their research blog that popular large language models like ChatGPT differ significantly from Orca-Math. Large language models are massive and complex, whereas Orca-Math, as a small language model, has relatively fewer parameters, typically in the thousands or millions, far lower than the billions or trillions of large language models. Additionally, Orca-Math focuses solely on solving math problems and cannot engage in conversations or answer other types of questions.

Despite having more parameters compared to other small language models, Orca-Math has 7 billion parameters, which is still much smaller than most well-known large language models. However, surprisingly, Orca-Math achieved a score of 86.81% in the GSM8K test, close to GPT-4-0613's 97.0%. In contrast, other models like Llama-2 performed poorly, with a score of only 14.6%.

Microsoft revealed that Orca-Math achieved such a high score because it utilizes higher quality training data compared to general large language models. Additionally, the Microsoft AI team developed an interactive learning process to continuously optimize results through teacher feedback. The Microsoft team concluded that small language models developed under specific conditions can perform comparably to large language models in certain applications.