Accuracy of Identifying AI-generated Academic Scientific Writing Exceeds 99%

2023-06-08

The debut of the AI chatbot ChatGPT has sparked global discussions due to its ability to generate text and engage in conversations that resemble human-like interactions. However, according to a study published on June 7th in the journal "Cell Reports-Physical Science," there are several indicators that can help us distinguish between AI chatbots and humans. Based on these indicators, researchers have developed a tool that can identify AI-generated academic scientific writing with an accuracy rate exceeding 99%. Heather Desaire, a professor at the University of Kansas and the first author of the research paper, stated, "We strive to create an easily understandable method that even high school students can use to establish an AI detector for different types of writing. Addressing the issue of AI writing is urgent, as people should be able to contribute to this field without needing a computer science degree." Desaire further explained, "Currently, AI writing has some very obvious issues. One of the biggest problems is that it aggregates text from various sources, making it difficult to perform accuracy checks. It's like playing the game of 'spot the difference.'" Although many AI text detectors are available online and perform quite well, they are not specifically designed for academic writing. To fill this gap, the research team aimed to build a more effective tool. They focused on a type of article called "perspective," which provides an overview of specific research topics written by scientists. The team selected 64 perspectives and created 128 articles generated by ChatGPT on the same research topic to train the model. When comparing these articles, they discovered indicators of predictability in AI writing. In contrast to AI, human paragraph structures are more complex, with varying sentence counts, total word counts, and sentence lengths. Punctuation and vocabulary preferences are also distinguishing factors. For example, scientists tend to use certain words like "however," "but," and "although," while ChatGPT frequently employs phrases like "other researchers" and "researchers." The team compiled a total of 20 features for the model to search for. During testing, the model achieved 100% accuracy in distinguishing articles entirely generated by AI from human-written articles. For identifying individual paragraphs within an article, the model achieved 92% accuracy. In similar tests, the research team's model far surpassed existing AI text detectors on the market. Next, the research team plans to determine the applicability of the model. They aim to test it on broader datasets and different types of academic scientific writing. As AI chatbots advance and become more sophisticated, researchers also want to assess the reliability of their model. "When people hear about this research, their first question is often, 'Can I use this to determine if my students actually wrote their papers?'" Desaire said. While the model excels at differentiating between AI and scientists, Desaire clarified that it was not designed to detect AI-generated student essays for educators. However, she pointed out that people can easily replicate their methods to establish models for their own purposes. Source: https://techxplore.com/news/2023-06-ai-generated-academic-science-accuracy.html

RECENT AI NEWS

RECENT AI TOOLS