Optimal quantitative metrics reflecting surgeons’ qualitative evaluation in deep learning-based intraoperative anatomy recognition task: a retrospective study

The study aimed to identify evaluation metrics for semantic segmentation in surgical navigation that best correlate with surgeons’ qualitative assessments. This single-center retrospective feasibility study focused on deep learning-based semantic segmentation of the ureter during laparoscopic sigmoid colectomy. To intentionally vary recognition performance, 20 deep learning models were developed by adjusting the training data and loss functions, resulting in diverse prediction images. Surgeons qualitatively evaluated these images for practicality, and subsequently used approximately half of these inference images to develop a new evaluation metric, Surgical Dice Similarity Coefficient. The remaining half was used to evaluate which of the seven metrics, including both the existing ones and the Surgical Dice Similarity Coefficient, showed the strongest correlation with the qualitative assessments. Overall, 22,294 images depicting the ureter were annotated, with up to 16,034 images used for training data and 6,260 for validation data, enabling the development of 20 deep learning models with intentionally varied performance. In total, 658 prediction images were evaluated to calculate the correlation coefficients between surgeons’ qualitative assessments and the models’ quantitative evaluations. Among the metrics, Surgical Dice Similarity Coefficient showed the highest correlation (0.625). This study introduces the Surgical Dice Similarity Coefficient, a novel evaluation metric that closely reflects surgeons’ qualitative perceptions in ureter segmentation. Using the Surgical Dice Similarity Coefficient as an evaluation metric in developing artificial intelligence models for surgical image recognition could help create more practical and effective models in surgery.