Replacing frontline employees with artificial intelligence is not a good idea.

Artificial intelligence chatbots have been widely used by businesses to greet customers and answer their questions over the phone or website. Some companies have found that to some extent, they can replace humans in call center roles.

However, existing evidence suggests that caution is needed and ethical oversight may be required when using these frontline tools in certain areas such as healthcare and human resources.

A recent high-profile example is a chatbot called Tessa used by the National Eating Disorder Association (NEDA) in the United States. The organization had a hotline operated by paid staff and volunteers to assist vulnerable individuals with eating disorders.

However, this year, the organization disbanded the hotline staff and announced its replacement with the Tessa chatbot. The reasons behind this decision are controversial. Former employees claim that the transition occurred after the hotline staff decided to form a union. The vice president of NEDA, on the other hand, believes that there was an increase in call volume and wait times, as well as legal liability concerns with using volunteer staff.

Regardless, after running for a short period of time, Tessa ceased to be used. Reports indicated that the chatbot provided problematic advice, which could exacerbate the symptoms of individuals seeking help for eating disorders.

According to reports, the two researchers involved in Tessa's creation, Dr. Ellen Fitzsimmons-Craft and Dr. C Barr Taylor, explicitly stated that the chatbot was not intended to replace the existing hotline or provide immediate help for those with severe eating disorder symptoms.

Significant Upgrades

So, what was the purpose of Tessa's design? The researchers conducted an observational study in collaboration with colleagues to elucidate the challenges they faced when designing a rule-based chatbot to interact with users concerned about eating disorders. It is an intriguing article that showcases design choices, operations, pitfalls, and corrections.

The initial version of Tessa was a traditional rule-based chatbot and also a highly mature chatbot. It followed a logic-based predefined structure and could not deviate from pre-programmed responses calibrated by its creators.

Their conclusions included the following viewpoint: "Rule-based chatbots have the potential to provide information and simple interactions to large populations at low cost, but are limited in understanding and appropriately responding to unexpected user answers."

This seemed to limit the applications for which Tessa was suitable. So, how did it replace the hotline previously used by NEDA? According to NPR, the company hosting the chatbot transformed Tessa from a rule-based chatbot with pre-programmed responses to a chatbot with "enhanced question and answer capabilities."

The later versions of Tessa adopted generative AI. These advanced AI chatbots can simulate human conversation patterns to provide more authentic and useful responses. Generating these customized responses relies on extensive information databases, and AI models "understand" through various technological processes such as machine learning, deep learning, and natural language processing.

Learning Lessons

Ultimately, the chatbot generated potentially harmful responses to some users' questions. The subsequent discussion shifted responsibility from one organization to another. However, the issue remains, and it could have been avoided if there was an institution providing ethical oversight, a "human in the loop," and adherence to Tessa's original explicit purpose.

In the context of integrating AI into various systems, it is crucial to learn from such cases. Although these events occurred in the United States, they have relevance for individuals in other countries seeking to do the same.

The UK seems to have a slightly more fragmented approach to this issue. Recently, the advisory board of the Center for Data Ethics and Innovation (CDEI) in the UK was dissolved and replaced by the newly formed Frontier AI Taskforce. There have also been reports of trials of AI systems as tools to assist workers in London, but not as replacements for hotlines.

Both of these examples highlight the potential conflict between ethical considerations and business interests. We must strike a balance that takes into account the interests of individuals and the efficiency and benefits that AI can provide.

However, in certain fields involving public interaction, AI-generated responses and simulated empathy may never be able to replace genuine human compassion, especially in the medical and mental health domains.