Why is ChatGPT getting dumber?

2023-08-08

Whether you have personally used ChatGPT or heard about it from elsewhere, the rumors are true: ChatGPT is getting dumber.

This phenomenon is quite perplexing because generative AI models are supposed to become smarter over time as they train themselves using user input, accumulating more user data.

The answer may lie in a concept called "drift."

"Drift" refers to the unexpected or unpredictable behavior exhibited by large language models (LLMs) in a way that deviates from the original parameters. This could be due to attempts to improve certain functionalities of complex AI models resulting in poorer performance in other aspects.

Researchers from the University of California, Berkeley and Stanford University conducted a study to assess the phenomenon of drift and examined the temporal changes in ChatGPT's popular LLMs, GPT 3.5 (the LLM behind ChatGPT) and GPT-4 (the LLM behind Bing Chat and ChatGPT Plus).

The study compared the capabilities of the two LLMs in solving math problems, answering sensitive questions, responding to surveys, generating code, taking medical licensing exams, and performing visual reasoning tasks in March and June.

From the research findings above, it can be observed that the March version of GPT-4 outperformed the June version in many aspects, most notably in basic math prompts, where the March version of GPT-4 performed better in both examples (a) and (b).

GPT-4 also performed worse in code generation, answering medical exam questions, and responding to surveys. All of these cases can be attributed to the phenomenon of drift.

About drift, one of the researchers, James Zou, told The Wall Street Journal, "We suspected this might be happening here, but we were surprised by how quickly the drift occurred."

Despite the decline in intelligence, GPT-4 and GPT-3.5 also have some improvements.

Therefore, the researchers encourage users to continue using LLMs but with caution and constant evaluation.

RECENT AI NEWS

RECENT AI TOOLS