What is the "Illusion" of Artificial Intelligence Chatbots

What are AI chatbots actually doing when they "hallucinate"? Does this term accurately capture why generative AI tools, which are almost ubiquitous, sometimes generate false information?

As the debate over the authenticity, capabilities, and trajectory of AI applications unfolds, a top expert in the field has challenged the concept of "hallucination," arguing that the term inaccurately describes how current AI models operate.

Usama Fayyad of Northeastern University said, "Overall, we don't like the term because these models make mistakes, and we can explain why they make mistakes."

Fayyad said the term "hallucination" was popularized by Google in response to the launch of OpenAI's influential ChatGPT. While the term captures to some extent the shortcomings of ChatGPT and similar products, it is somewhat inappropriate here and may negatively impact the public's understanding of AI technology.

Fayyad continued, "When you say hallucination, you're attributing too much to the model. You're attributing intent to it; you're attributing consciousness to it; you're attributing some kind of understanding on the machine side."

Fayyad emphasized that chatbots "have no intent; no understanding."

There is still no accurate data on the frequency of chatbots "hallucinating," although some companies have been working to quantify the error rates of widely used large language models. According to The New York Times, a startup called Vectra, founded by former Google employees, found that OpenAI's model hallucinates about 3% of the time, while Google's platform called "Chat on the Go" has a false information rate of 27%.

Fayyad said that what makes it more complex is that the output generated by current generative AI models is highly dependent on prompts. A slight adjustment to the prompt can result in a very different outcome.

Byron Wallace, who has referred to these prompts as "not entirely prompt engineering," described the process of designing inputs for chatbots as more like "incantation and black magic."

Fayyad aims to reduce the variability in prompts and simplify the conversation surrounding the potential applications of generative AI tools.

Fayyad said, "I can say - these models hallucinate; or, more accurately, the models make a mistake, and we know that these models make mistakes."

Fayyad proposes a need for greater trust between humans and AI machines. He believes that "practitioners, users, and organizations need to trust the system in terms of how it makes decisions, how it operates, and that it won't exhibit any erratic, unpredictable, or dangerous behavior."

He wrote, "The topic of AI breeds mystery and ambiguity." "Unraveling the mystery of the behaviors exhibited by technology and algorithms, for better or worse, will lead to true progress in theory, academia, business, and practice, and create valuable outcomes."

Currently, large language models like ChatGPT are "advanced autocomplete" applications trained on vast amounts of digital text from online databases, articles, and other sources. "They're just producing output, much like your phone or any other autocomplete device."

"These models don't know the difference between the correct sequence and the wrong one," Fayyad said. "Understanding where the errors occur and attempting to recover from them - that's a very challenging AI problem, and we don't have good solutions for it today."

To control hallucination, researchers have started using other large language models to check the accuracy of various chatbots. However, these tools can also produce errors (hallucinations), Fayyad pointed out.

He emphasized that humans need to continue inspecting the output generated by these tools - a concept known as "humans in the loop."

He said, "It will be up to you as the user to judge, 'This autocomplete is not correct,' and then correct it."