Diagnostic Performances of GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro in “Diagnosis Please” Cases

Abstract Backgrounds Large language models (LLMs) are rapidly advancing and demonstrating high performance in understanding textual information, suggesting potential applications in interpreting patient histories and documented imaging findings. LLMs are advancing rapidly and an improvement in their diagnostic ability is expected. Furthermore, there has been a lack of comprehensive comparisons between LLMs from various manufacturers. Purpose We tested the diagnostic performance of the latest three major LLMs (GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro) using Radiology Diagnosis Please cases, a monthly diagnostic quiz series for radiology experts. Materials and Methods Clinical history and imaging findings as provided textually by the case submitters were extracted from 324 quiz questions from Radiology Diagnosis Please cases. The GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro generated the top three differential diagnoses. Diagnostic performance among the three LLMs were compared using Cochrane’s Q and post-hoc McNemar’s tests. Results The diagnostic accuracies for the primary diagnosis were 41.0%, 54.0%, and 33.9% for GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro, respectively. When considering the accuracy of any of the top three differential diagnoses, the rates improved to 49.4%, 62.0%, and 41.0%, respectively. Significant differences in diagnostic performance were observed among all pairs of the models. Conclusion In a comparison of the latest LLMs, Claude 3 Opus outperformed GPT-4o and Gemini 1.5 Pro in solving radiology quiz cases. These models appear capable of assisting radiologists when supplied with accurate evaluations and worded descriptions of imaging findings by radiologists. Summary statement Claude 3 Opus achieved the highest diagnostic accuracy, followed by GPT-4o and Gemini 1.5 Pro, in a comparison of their performance on 324 text-based Radiology Diagnosis Please cases.. Key Results This study compared the diagnostic performances of the latest three major large language models, GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro, using clinical history and textualized imaging findings in Radiology Diagnosis Please cases. The top three differential diagnoses generated by GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro achieved diagnostic accuracies of 49.4%, 62.0%, and 41.0%, respectively, with statistically significant differences between each model’s performance.