Exploring the Ability of Google's ChatGPT in Solving Mathematical Problems

2023-08-29

OpenAI's chatbot ChatGPT performs well in many tasks such as script writing, explaining complex topics, debugging, and code interpretation, but it performs poorly in mathematics.


Recently, a research paper by Stanford University and the University of California, Berkeley, pointed out that large language models (LLMs) can perform simple mathematical operations with small numbers, but struggle with large numbers. This indicates that LLMs have not learned the basic rules required to perform these arithmetic operations. The research paper further states that even with improvements made to GPT-4 using the MATHS dataset, errors mainly occur due to arithmetic and calculation mistakes.


Competitor Google has recognized the issues with LLMs and has started teaching models like ChatGPT to better reason algorithms. The work of Google researchers titled "Teaching Language Models to Reason Algorithmically" adopts a context learning approach and introduces a better reasoning algorithm.


Context learning refers to the process of guiding models in teaching new skills through gradual guidance, rather than providing all instructions at once. This approach refers to the ability of the model to perform tasks after seeing some examples in context.


They also propose a prompting technique to make general language models have strong generalization capabilities for more challenging mathematical problems. This technique builds upon other rational reinforcement-based methods such as scratch paper and thinking chains. Finally, they demonstrate that the model can "reliably execute algorithms beyond the distribution of training examples" with the correct prompt.


Below-average students


ChatGPT has become worse at performing certain basic mathematical operations while improving in other aspects. The same study points out that the performance of this highly anticipated chatbot has been declining compared to its early performance in March.


Researchers attribute this deterioration to an artificial intelligence phenomenon called drift, where attempting to improve one part of a complex model can make other parts worse.


To track performance, Stanford University professor James Zou and his colleagues Mattia Zorzi and Linjiao Chen inputted 1,000 different numbers to ChatGPT. In March, the paid GPT-4 version impressively correctly identified 84% of the numbers as prime or non-prime. By June, the success rate dropped to 51%.


In addition to answering questions incorrectly, ChatGPT has also faced criticism for attempting to show its reasoning process to researchers. As part of the study, researchers asked the chatbot to explain its "thought chain," which refers to the terms used by the chatbot to explain its reasoning process. In March, it was able to do so, but by June, it stopped displaying its step-by-step reasoning.


Recent Google research attempts to address this issue through its context learning approach. These findings suggest that exploring longer contexts and prompting more informative explanations may provide valuable research outcomes.


Wolfram Principal



Wolfram Research is a pioneer in combining technology with mathematical education. They collaborated with OpenAI, the parent company of ChatGPT, to bring stronger mathematical capabilities to AI models. Conrad Wolfram, co-founder of the company, revealed in an interview with AIM, "We've had some interesting results with our LLM module. I tried running a UK high school math exam question, which is a pre-university exam that students take, and with just ChatGPT, we got 43% accuracy, which is quite impressive. But with the Wolfram module added, the accuracy reached 96%."


He jokingly said, "That's game over for humans."


It is worth noting that when the same mathematical problem was asked to ChatGPT versions 3.5, 4, and the Wolfram plugin separately, only the Wolfram plugin provided the correct answer on the first attempt.


The Wolfram+ChatGPT plugin not only solves mathematical problems step by step but also presents the answers in a visual manner with special prompts. Based on the prompts, it can further present data in the form of charts, curves, and histograms.


The plugin can convert natural language queries into beautiful mathematical equations. It is able to do so by combining ChatGPT's imitation of human conversation techniques with Wolfram's strong foundation in symbolic programming languages, which focuses on expressing thoughts in computational form.


While Wolfram's plugin has made significant progress, researchers have found that the model's performance is deteriorating. In the current technological environment, Google's latest context learning approach can help AI chatbots become excellent students.


RECENT AI NEWS

RECENT AI TOOLS