Automated Segmentation of Mass Lesions in Mammography: Mask R-CNN Versus Classical Methods, Fuzzy Clustering and MedSAM

Background/Objectives: The automatic and accurate segmentation of mass lesions in mammograms is a critical step in the computer-aided diagnosis workflow. This study evaluates the mask region-based convolutional neural network (Mask R-CNN), a robust deep learning model for instance-level segmentation, highlighting its growing potential in medical imaging. Methods: Following configuration optimization, the model is applied to mammography images from the CSAW-CC and TOMPEI-CMMDs, which provide masks annotated by breast radiology experts, to be used as a ground truth. To quantify shape and contour discrepancies with respect to the radiologists’ annotations, performance is evaluated with both overlap- and distance-based metrics. Furthermore, the model is tested on synthetic mammograms and benchmarked against classical segmentation methods (Otsu thresholding, gradient vector flow active contour, and Chan–Vese segmentation), Fuzzy c-means clustering, and the state-of-the-art Medical Segment Anything (MedSAM) model. Results: By leveraging transfer learning, a limited dataset suffices for stable training, with the network converging by the 10th epoch. The optimal configuration achieves a bounding box average precision (BBox AP) of 0.80 and a segmentation average precision (Segm AP) of 0.58. Evaluated on a test set of real mammograms, the model correctly identifies the lesions, yielding an average Dice similarity coefficient (DSC) of 0.87 and an average intersection over union (IoU) of 0.77, performance aligned with the current state of the art. Conclusions: Mask R-CNN has emerged as a reliable and robust model for breast lesion segmentation; compared to the other methods evaluated in this study, it demonstrates superior performance across both overlap- and distance-based metrics. The model exhibits high DSC and IoU scores even on synthetic mammograms with a priori known lesion profiles, which can be used as an objective ground truth to mitigate annotation bias from inter-operator variability among radiologists.