Stack Overflow Regains Dominance from ChatGPT

2023-08-18

Since 2008, whenever a programmer has a problem, they would first go to Stack Overflow (SO). That is until OpenAI released ChatGPT.

ChatGPT has been very useful in information retrieval. However, new research suggests that this highly anticipated chatbot may not be the best solution for software engineering prompts. Just like programming questions on SO, OpenAI's ChatGPT is wrong more than half of the time in the same context.

Purdue University conducted a detailed study to determine the effectiveness of ChatGPT as there is no data showing how much help it provides in answering these types of prompts. Researcher Samia Kabir and her team presented 517 questions similar to those on SO to ChatGPT. The team studied the accuracy and quality of these answers.

The research results tell a compelling story. Out of all the answers, 52%, or 259 answers, were incorrect, while 48% were accurate. Additionally, 77% of the answers were verbose. This staggering number of incorrect responses, although seemingly clear, raises concerns about potential impacts on clarity and efficiency. Paradoxically, the observations in the research paper imply that the errors of the AI model lie in its inaccuracy.

One user also stated that in their experience, ChatGPT mostly produces incorrect answers when prompted on well-known topics. "Whether it's correct, incorrect, or completely wrong, it generates equally confident language. Therefore, on topics I don't know, it is very likely to produce confidently wrong answers. I cannot determine if the text it generates is approximately correct, extremely wrong, or just slightly inaccurate. So, it is obviously worse than useless."

Full Support for Semantics

In a study titled "Who Answers Better? A Deep Analysis of ChatGPT and Stack Overflow for Software Engineering Questions," researchers also found that OpenAI's ChatGPT is more prone to conceptual errors rather than factual errors. The study points out, "Many answers are incorrect because ChatGPT fails to understand the underlying context of the questions asked."

Earlier this month, SO decided to adopt semantic search as the traffic to the page kept growing. In an official blog post, the company stated, "Semantic search and LLM go together like cookies and milk." In layman's terms, semantic search can understand the meaning and intent of queries in a human-like manner. Therefore, it can provide precise and contextually relevant search results.

SO further stated in their official blog that their "mission is simple: accuracy and attribution." Whereas existing GPT models generate results from unknown sources, the company has taken steps to provide attribution for the questions and answers used in the Retrieval-Augmented Generation (RAG) LLM summaries.

Upgrading with AI

The impact of the research goes beyond the performance of ChatGPT.

Since the release of ChatGPT, news of this AI chatbot replacing Stack Overflow has spread. This news is based on the decline in the number of users on the developer Q&A platform. According to Purdue's research, the declining user count on traditional platforms like SO indicates a reshaping of the landscape for online programming assistance due to the popularity of ChatGPT.

This shift was confirmed in the 2023 Stack Overflow Annual Developer Survey, which gathered insights from 90,000 developers. The survey also emphasized that 77% of developers have a positive view of AI tools. However, only 42% of developers trust these tools in terms of accuracy. To reverse this trend, the New York-based company launched a series of AI tools named OverflowAI two weeks ago.

As a strategic response to SO, it also launched GenAI Stack Exchange, a dedicated community platform for exchanging insights on AI tools. These recent initiatives reflect SO's conscious efforts to adapt to developers' preferences for seeking AI knowledge. Additionally, SO introduced Stack Overflow Natural Language Processing (NLP) Consortium and introduced a feature called Discussions for engaging in AI debates around subtle differences in technical approaches.

By launching these latest AI features, the company is striving to compete fiercely with the currently popular tool, ChatGPT. Despite a slight decline, Purdue's research concludes that SO maintains an advantage in the engineering department.

RECENT AI NEWS

RECENT AI TOOLS