Assessing the Feasibility of a Multimodal Annotation and Segmentation System to Detect Carious Lesions From Intraoral Photographs

Purpose To develop and validate a multimodal annotation workflow for identifying carious and non‐carious lesions using intraoral photographs, and to assess the effects of confidence thresholds on model performance. Methods A multimodal system was developed integrating the YOLOv8, Gemini Flash 2.5 Large Language Model, Segment Anything Model (SAM) and the Streamlit Python Framework. Clinical intraoral photographs from patients with complex dental needs were obtained and augmented to 6489 images. Model performance was evaluated using precision, recall, Kappa agreement, mean average precision at an Intersection‐over‐Union (IoU) of 50% (mAP50), mAP50‐95 and precision variations at 25%, 50% and 75% confidence thresholds. Results The resulting performance metrics were a mAP50 for bounding boxes of 97.58%, a mAP50 for masks of 92.46%, a mAP50‐95 for bounding boxes of 85.4% and a mAP50‐95 for masks of 68.88%. Misclassification rates also varied by feature and confidence threshold. Staining showed a misclassification rate of 9.7% at 0.25 confidence, 10.4% at 0.50 confidence and 46.6% at 0.75 confidence on the test sets. Similarly, micro‐cavitation demonstrated misclassification rates of 12.5%, 25.8% and 56.0% at the respective confidence thresholds of 0.25, 0.50 and 0.75. Validation on a public dataset achieved an mAP50 of 79% and an mAP50‐95 of 64.8%. Conclusion An in‐house multimodal object detection system achieved reasonable precision in carious and non‐carious lesion detection, with 50% confidence yielding optimised prediction consistency.