Evaluating multi-task network architectures for simultaneous breast lesion segmentation and classification in ultrasound images

Abstract Breast lesion segmentation and classification in ultrasound (US) images are two essential tasks for computer-aided diagnosis of breast cancer. However, these tasks are still challenging, mainly due to the high variability of lesions and the poor image quality of US. Numerous deep learning methods have been proposed to assist physicians in performing breast lesion segmentation and classification. Considering that these tasks are related to common features, learning them jointly through multi-task learning (MTL) represents a viable approach to improve the performance of each individual task. In this paper, we present and compare multiple MTL network configurations for the simultaneous segmentation and classification of breast lesions in ultrasound images. Building on two state-of-the-art architectures, namely SegResNet for segmentation and EfficientNet for classification, we designed and evaluated several combined configurations of these models arranged in different MTL schemes. These configurations explore various levels of feature sharing and integration between the tasks, aiming to identify the most effective architectural arrangement for joint lesion segmentation and malignancy classification in breast ultrasound. Moreover, these configurations are also compared with state-of-the-art MTL configurations. Experimental results based on a dataset compiled from two different centers, comprising 810 2D breast US images, indicate that combining SegResNet and EfficientNet in a MTL configuration achieved accurate performance in both segmentation and classification. Among the tested configurations, the best-performing model combined a segmentation network with an EfficientNet classification network that utilized shared encoded features from the SegResNet, achieving a Dice coefficient of 81.19% for lesion segmentation and an area under the ROC curve (AUC) of 97.27% for lesion classification. Overall, the proposed networks demonstrated their potential in enhancing computer-aided diagnosis of breast cancer. Graphical abstract