Prompt-based evaluation of ChatGPT and Gemini for mandibular fracture detection and localization on panoramic radiographs

Background Multimodal large language models (MLLMs) can process visual and textual inputs; however, their clinical relevance in maxillofacial trauma imaging remains uncertain. This study evaluated the prompt-based performance of ChatGPT and Gemini for mandibular fracture detection and localization on panoramic radiographs and compared their performance with that of a blinded oral and maxillofacial surgeon. Methods This retrospective image-based study included 104 archived pre-treatment panoramic radiographs: 64 with mandibular fractures and 40 no-fracture controls. Reference fracture status and anatomical localization were determined retrospectively from available institutional clinical, radiographic, treatment, operative, and pre-treatment CBCT/CT records. ChatGPT and Gemini assessed each radiograph three independent times using standardized prompts for fracture detection and localization. Majority voting was used for fracture detection, whereas an ordinal consensus score across repetitions was used for localization. A blinded oral and maxillofacial surgeon, who was unaware of the study aim and reference diagnoses, independently evaluated all radiographs and recorded free-text radiographic findings. Diagnostic performance measures were reported with 95% confidence intervals, and paired comparisons were adjusted using the Holm procedure. Results The blinded expert correctly classified all 104 radiographs, achieving 100% sensitivity, specificity, and accuracy. Gemini achieved a sensitivity of 92.2%, specificity of 65.0%, and accuracy of 81.7%, whereas ChatGPT achieved corresponding values of 35.9%, 97.5%, and 59.6%. Overall detection performance differed significantly among the three methods (Cochran’s Q, p < 0.001), and Gemini achieved significantly higher overall classification accuracy than ChatGPT (Holm-adjusted p = 0.0018). For fracture localization, fully correct responses were obtained in 62 of 64 cases (96.9%) by the expert, 10 of 64 cases (15.6%) by Gemini, and 17 of 64 cases (26.6%) by ChatGPT. Ordinal localization scores differed significantly among the methods ( p < 0.001); the expert scored higher than both models, and Gemini scored higher than ChatGPT because it produced more partially correct responses, whereas ChatGPT achieved exact-match localization more often (26.6% vs. 15.6%). Conclusions The blinded human expert substantially outperformed both MLLMs in mandibular fracture assessment. Gemini showed higher sensitivity and broader but less precise localization behavior, whereas ChatGPT showed higher specificity but missed a larger proportion of fractures. These findings indicate that the prompt-based, text-generating responses of general-purpose MLLMs remain inconsistent and should not be considered substitutes for expert clinical and radiological assessment.