Enhancing YOLOv8s with MHSA and GA for robust caries detection on periapical radiographs

Background Early identification of dental caries on Intraoral Periapical Radiographs is clinically important, yet automated detectors often show unstable performance across datasets due to subtle lesion appearance and limited generalization. Methods We propose an attention-augmented YOLOv8s detector that inserts two Multi-Head Self-Attention (MHSA) blocks in the deeper backbone and couples this architecture with a compact Genetic Algorithm (GA)–based schedule search to stabilize optimization. Experiments were conducted on a curated dataset collected at Sibar Institute of Dental Sciences, Guntur (1,887 radiographs), standardized and quality screened, and split into 80% for training, 10% for validation, and 10% for testing. Results Using identical augmentations and a fixed inference operating point, the proposed YOLOv8s+MHSA+GA achieves Precision 0.98, Recall 0.94, F1 Score 0.96, mAP@0.5 0.97, and mAP@0.5:0.95 0.94. Relative to the next-best baseline (per metric), this corresponds to +0.21 mAP@0.5 (0.97 vs 0.76), +0.61 mAP@0.5:0.95 (0.94 vs 0.33), and +0.26 Recall (0.94 vs 0.68). We further report Wilson 95% confidence intervals for precision and recall to quantify statistical uncertainty. Conclusions These results indicate that deeper-stage self-attention, paired with principled schedule selection, delivers clinically meaningful gains in sensitivity and localization tightness while preserving the efficiency advantages of a one-stage detector, supporting routine AI-assisted diagnosis.