Performance of GPT-4 in Oral and Maxillofacial Surgery Board Exams: Challenges in Specialized Questions

Abstract Purpose The aim of this study was to evaluate the performance of GPT-4 in answering oral and maxillofacial surgery (OMFS) board exam questions, given its success in other medical specializations. Methods A total of 250 multiple-choice questions were randomly selected from an established OMFS question bank, covering a broad range of topics such as craniofacial trauma, oncological procedures, orthognathic surgery, and general surgical principles. GPT-4's responses were assessed for accuracy, and statistical analysis was performed to compare its performance across different topics. Results GPT-4 achieved an overall accuracy of 62% in answering the OMFS board exam questions. The highest accuracies were observed in Pharmacology (92.8%), Anatomy (73.3%), and Mucosal Lesions (70.8%). Conversely, the lowest accuracies were noted in Dental Implants (37.5%), Orthognathic Surgery (38.5%), and Reconstructive Surgery (42.9%). Statistical analysis indicated significant variability in performance across different topics, with GPT-4 performing better in general topics compared to specialized ones. Conclusion GPT-4 demonstrates a promising ability to answer OMFS board exam questions, particularly in general medical topics. However, its performance in highly specialized areas reveals significant limitations. These findings suggest that while GPT-4 can be a useful tool in medical education, further enhancements are needed for its application in specialized medical fields.