An evaluation of large language models for panoramic radiographs

Conversational multimodal models have demonstrated the ability to interpret medical images directly, but their performance on panoramic radiographs has not been systematically evaluated against dental reference standards. We systematically evaluated their ability to identify radiographic findings and localise the affected teeth. Reference labels were established by two experienced dentists, with disagreements resolved by consensus and, when necessary, adjudicated by a senior dentist. We evaluated seven models under zero-shot and four-shot conditions, with three runs per condition. We assessed seven binary tasks and three FDI localisation tasks using macro-averaged accuracy, valid-response rate, conditional F1, and FDI F1, and reported sensitivity, specificity, positive predictive value, and negative predictive value with 95% confidence intervals. The GPT-5.4 family achieved the strongest overall image-level performance, although the leading model varied by task and prompting condition. GPT-5.4-mini achieved the highest four-shot FDI F1. Zero-shot macro-averaged accuracy was highest for GPT-5.4-mini (0.774), and four-shot accuracy was highest for GPT-5.4 (0.778). Compared with zero-shot prompting, four-shot prompting increased macro-averaged accuracy in six of seven models, with a maximum gain of 25.0 percentage points; GPT-5.4-mini decreased by 2.9 percentage points. GLM-4.6V-Flash had the largest increase, accompanied by a 0.383 increase in valid-response rate but a 0.122 decrease in conditional F1. Conditional F1 decreased in six models, and no condition had mean FDI F1 above 0.185. Four-shot prompting improved image-level accuracy in most models, but tooth-level localisation remained poor. This benchmark shows that improved image-level accuracy with four-shot prompting does not ensure reliable tooth-level localisation.