YOLOv8 Outperforms Traditional CNN Models in Mammography Classification: Insights From a Multi‐Institutional Dataset

ABSTRACT This study evaluates the efficacy of four deep learning methods—YOLOv8, VGG16, ResNet101, and EfficientNet—for classifying mammography images into normal, benign, and malignant categories using a large‐scale, multi‐institutional dataset. Each dataset was divided into training and testing groups with an 80%/20% split, ensuring that all examinations from the same patient were consistently allocated to the same split. The training set for the malignant class contained 10 220 images, the benign class 6086 images, and the normal class 8526 images. For testing, the malignant class had 1441 images, the benign class 1124 images, and the normal class 1881 images. All models were fine‐tuned using transfer learning and standardized to 224 × 224 pixels with data augmentation techniques to improve robustness. Among the models, YOLOv8 demonstrated the highest performance, achieving an AUC of 93.33% for the training dataset and 91% for the testing dataset. It also exhibited superior accuracy (91.82% training, 86.68% testing), F1‐score (91.11% training, 84.86% testing), and specificity (95.80% training, 93.32% testing). ResNet101, VGG16, and EfficientNet also performed well, with ResNet101 achieving an AUC of 91.67% (training) and 90.00% (testing). Grad‐CAM visualizations were used to identify the regions most influential in model decision‐making. This multi‐model evaluation highlights YOLOv8's potential for accurately classifying mammograms, while demonstrating that all models contribute valuable insights for improving breast cancer detection. Future clinical trials will focus on refining these models to assist healthcare professionals in delivering accurate and timely diagnoses.