"GPT-4 shines in the field of medicine, can it become a reliable assistant for doctors?"

2023-08-15

OpenAI's capabilities have permeated various fields, but there is one field where, if LLM is used correctly, it can directly impact people's lives and have the greatest influence - the medical field. Earlier this year, ChatGPT even passed all three parts of the United States Medical Licensing Examination (USMLE), and we have even seen how ChatGPT helped save a dog's life through accurate medical diagnosis. However, we haven't seen much practical application in the medical field. Does the capability of GPT-4 make it a suitable player in the medical field?


Huge Potential


OpenAI and Microsoft released a paper in March this year on the capabilities of GPT-4 in tackling medical challenges. In this study, GPT-4 demonstrated impressive language understanding and generation abilities in the medical field. The study evaluated the performance of GPT-4 using medical exams and benchmark datasets, even though the model was not specifically optimized for the medical domain.


Researchers used official USMLE practice materials and the MultiMedQA dataset to evaluate the performance of GPT-4. GPT-4 exceeded the passing score of USMLE by more than 20 points, surpassing previous models (including GPT-3.5) and models specifically fine-tuned for medical knowledge. Additionally, GPT-4 demonstrated better probability calibration, indicating its proficiency in predicting correct answers. The study also explored how GPT-4 interprets medical reasoning, custom explanations, and creates hypothetical scenarios, showcasing its potential in medical education and practice. These findings highlight the capabilities of GPT-4 while acknowledging the challenges related to accuracy and safety in real-world applications.


Compared to its previous versions, GPT-4 has shown significant improvement in official medical exams like USMLE. It outperformed GPT-3.5 by more than 30 percentage points. While GPT-3.5 approached the passing score (60% correct for multiple-choice questions), GPT-4 passed with a significant advantage.


When comparing the early version of GPT-4 (referred to as the base model) with GPT-4, the former performed slightly better on certain tests, with an improvement of about 3-5%. This suggests that in the process of making the model safer and better at following instructions, it may lose some of its original performance. Researchers suggest that future work can seek more effective ways to balance accuracy and safety by improving the training process or using specialized medical data.


Where is Med-PaLM?


The aforementioned study did not compare GPT-4 with models like Med-PaLM and Flan-PaLM 540B because these models were not open to the public during the study.


Google recently introduced their multimodal medical LLM, called Med-PaLM - a large-scale multimodal generation model that can encode and interpret biomedical data. Its capabilities are more advanced than GPT-4 as it can handle various types of medical data such as clinical language, medical images, genomics, and even perform a wide range of tasks. The model can generalize to new medical tasks and perform multimodal reasoning without specific training. It accurately identifies and interprets medical conditions in images based on natural language instructions and prompts.


Not Without Flaws


However, the applications of GPT-4 are not as diverse as those provided by Med-PaLM. While GPT-4 is announced to have multimodal capabilities, it is not yet available to users. Furthermore, there have been negative observations regarding the ability of GPT-4 in medical diagnosis. Issues and biases are part of the results, and there are concerns that GPT-4 tends to embed social biases, which may affect its clinical decision-making.


GPT-4 still suffers from the common problem of generating erroneous information. The model generated incorrect answers in medical citations. GPT-4 produced over 20% incorrect medical citations.


21% of the medical journal articles cited by GPT-4 were found to be fake; an estimated 98% of those cited by GPT-3.5 were fake. The narrower the topic, the more fake articles were generated. Despite its potential, ChatGPT is currently not a reliable source of medical data.


Although GPT-4 may not be a fully reliable tool for medical diagnostic assistance, it has other functionalities that can be helpful. Hospitals are seeking AI to help alleviate the burden on doctors. AI can facilitate smooth processes by applications that can write notes for electronic health records and draft empathetic notes for patients. Transcribing comments from doctors and patients and creating summaries for electronic health records is one of the best use cases in the medical field. Given the current limitations, GPT-4 still has a long way to go before being fully adopted in the medical field.

RECENT AI NEWS

RECENT AI TOOLS