Comparative Analysis of Third Molar Segmentation Performance Between Sexes Using Deep Learning Models

Background/Objectives: Sex analysis in dental radiographs plays a central role in forensic identification, especially when biological material is compromised or incomplete. While most AI-based studies rely on complete dentition or craniofacial structures, this study investigates whether sex-based information can be extracted solely through segmentation of third molars in panoramic radiographs. Methods: A retrospective dataset containing 2818 third molar annotations from 757 panoramic images with balanced class distribution across training, validation, and testing subsets was constructed. Three sample segmentation-based deep learning models—YOLOv12n, YOLO26n, and RT-DETR v2—were evaluated under the same training conditions using detection-focused metrics including sensitivity, recall, and mAP. Results: YOLOv12n demonstrated the most balanced performance, achieving the highest mAP@0.50 score of 0.810 and mAP@0.50–0.95 score of 0.574; RT-DETR v2 showed higher sensitivity but lower localization accuracy and significantly longer training time. YOLO26n yielded the highest recall rate but showed an increase in false positives. Class-based analyses indicated sex-specific morphological variability in third molar anatomy, showing consistently higher detection performance in female samples. Conclusions: These results demonstrate that isolated third molars encode distinctive sex-related signals and that segmentation-focused frameworks offer an interpretable and clinically relevant alternative to whole-image classification in forensic dentistry. Future studies should incorporate larger multi-population datasets, multi-tooth integration, and explainable AI techniques to further improve robustness and applicability.