The updated version of ChatGPT-4 AI chatbot has shown improved performance in answering multiple-choice practice questions in OphthoQuestions, a commonly used resource for ophthalmic certification. This research comes from a team at the University of Toronto, whose earlier studies revealed that the previous version of the chatbot correctly answered 46% of multiple-choice questions in January 2023, which increased to 58% in February 2023.
"Compared to the previous version, the updated version of this chatbot seems to have improved performance across all question categories in OphthoQuestions," wrote Dr. Rajeev H. Muni and his team, who lead the Department of Ophthalmology and Vision Sciences at the University of Toronto. "The research findings also indicate that the updated version of the chatbot is able to generate accurate answers in most cases when given options."
Evidence suggests that the AI chatbot produces human-like responses as a dynamic language model, aiming to enhance existing conversational AI systems. Since the previous version of ChatGPT correctly answered nearly half of the multiple-choice questions used for the American Board of Ophthalmology exam preparation, this analysis updates the results by evaluating the accuracy of the updated chatbot. The team inputted the same practice questions from the free trial version of OphthoQuestions, which were used in their earlier research on ophthalmic knowledge assessment for the Ophthalmic Knowledge Assessment Program (OKAP) and Written Qualifying Examination (WQE) tests, into ChatGPT-4 (released in March 2023 by OpenAI).
Muni and his colleagues recorded the proportion of ophthalmology trainees using the OphthoQuestions trial version who selected the same answers as the ChatGPT chatbot. The primary outcome of the study was the number of multiple-choice questions the chatbot answered correctly. The researchers used Microsoft Excel for data analysis, and the chatbot generated answers for the certification exam in March 2023.
The average length of the questions was 306.40 characters, while the average length of the chatbot's responses was 473.83 characters. The analysis yielded highly positive results: out of 125 text-based multiple-choice questions, the chatbot answered 105 correctly (84%). Additionally, the chatbot answered all questions correctly in the fields of general internal medicine, retina and vitreous, and uveitis (100%), but answered 8 out of 13 questions (62%) correctly in clinical optics.
On average, approximately 71% (95% CI, 66-75) of ophthalmology trainees selected the same multiple-choice answers as the chatbot. The researchers noted that the chatbot provided explanations and additional insights for 123 out of 125 questions (98%). When options for multiple-choice questions were removed, the analysis showed that the chatbot provided correct answers for 49 out of 78 individual questions (63%).
The median length of the correctly answered multiple-choice questions was 217 characters, while the incorrectly answered questions had a median length of 246 characters. The median length of the correctly answered questions was 428 characters, compared to 465 characters for the incorrectly answered questions.
Muni and his colleagues pointed out the limitations of this study, as the chatbot provided preparation materials for the certification exam, which may differ from the official exam. They also highlighted that the chatbot provides unique answers for each user, and repeating the study may yield different results.
"Previous research may have trained the chatbot under different settings," the researchers wrote. "The results of this study must be interpreted in the context of the study date, as the chatbot's knowledge base may continue to expand rapidly."
Source: https://www.hcplive.com/view/artificial-intelligence-chatbot-appears-improve-ophthalmic-knowledge-assessment
Alphabet Q1 paid subscribers reached 350M, fueled by YouTube & Google One. YouTube ad revenue slightly missed at $9.88B amid Premium growth. Total Q1 revenue hit $109.9B, beating forecasts.
Anthropic releases Claude connectors for Adobe, Blender, and Ableton, enabling AI workflow automation and app control. It also sponsors the Blender Development Foundation.
Apple introduces an App Store model splitting 12-month plans into monthly installments. On iOS 26.4+, it lowers upfront costs without reducing total price, adding billing tracking.
Google launches Aadhaar digital IDs in Wallet for India, enabling secure QR/NFC verification. A phased partner rollout targets key sectors, while passport-based credentials expand globally.
YouTube tests "Ask YouTube," an AI tool blending text and video for complex queries. Initially for US Premium users, it supports interactive follow-ups, expanding Google's AI search.
Google grants US DoD classified AI access, joining OpenAI and xAI in defense deals amid Anthropic's dispute. Contractual safeguards against surveillance and lethal weapons remain unenforceable.
Mistral AI launches Workflows, a Temporal-powered orchestration platform for secure, Python-driven enterprise AI automation featuring human-in-the-loop approvals.