Male pelvic multi‐organ segmentation on transrectal ultrasound using anchor‐free mask CNN

Purpose Current prostate brachytherapy uses transrectal ultrasound images for implant guidance, where contours of the prostate and organs‐at‐risk are necessary for treatment planning and dose evaluation. This work aims to develop a deep learning‐based method for male pelvic multi‐organ segmentation on transrectal ultrasound images. Methods We developed an anchor‐free mask convolutional neural network (CNN) that consists of three subnetworks, that is, a backbone, a fully convolutional one‐state object detector (FCOS), and a mask head. The backbone extracts multi‐level and multi‐scale features from an ultrasound (US) image. The FOCS utilizes these features to detect and label (classify) the volume‐of‐interests (VOIs) of organs. In contrast to the design of a previously investigated mask regional CNN (Mask R‐CNN), the FCOS is anchor‐free, which can capture the spatial correlation of multiple organs. The mask head performs segmentation on each detected VOI, where a spatial attention strategy is integrated into the mask head to focus on informative feature elements and suppress noise. For evaluation, we retrospectively investigated 83 prostate cancer patients by fivefold cross‐validation and a hold‐out test. The prostate, bladder, rectum, and urethra were segmented and compared with manual contours using the Dice similarity coefficient (DSC), 95% Hausdorff distance (HD 95 ), mean surface distance (MSD), center of mass distance (CMD), and volume difference (VD). Results The proposed method visually outperforms two competing methods, showing better agreement with manual contours and fewer misidentified speckles. In the cross‐validation study, the respective DSC and HD 95 results were as follows for each organ: bladder 0.75 ± 0.12, 2.58 ± 0.7 mm; prostate 0.93 ± 0.03, 2.28 ± 0.64 mm; rectum 0.90 ± 0.07, 1.65 ± 0.52 mm; and urethra 0.86 ± 0.07, 1.85 ± 1.71 mm. For the hold‐out tests, the DSC and HD 95 results were as follows: bladder 0.76 ± 0.13, 2.93 ± 1.29 mm; prostate 0.94 ± 0.03, 2.27 ± 0.79 mm; rectum 0.92 ± 0.03, 1.90 ± 0.28 mm; and urethra 0.85 ± 0.06, 1.81 ± 0.72 mm. Segmentation was performed in under 5 seconds. Conclusion The proposed method demonstrated fast and accurate multi‐organ segmentation performance. It can expedite the contouring step of prostate brachytherapy and potentially enable auto‐planning and auto‐evaluation.