Importance of the Quality of Annotation: Impact of Simulated Inter-Observer Variability on Deep Neural Network Automated Segmentation Model Performance

Background: Deep neural network based prostate segmentation depends on manual annotations, yet the effect of annotation variability on model performance remains underexplored. Methods: Prostate contours were manually delineated by an expert clinician on 119 T2-weighted MR images from the PROSTATEx Challenge 2017 training dataset, and slice-wise synthetic radial modifications of 1–10 mm were applied to create 10 modified training datasets plus an unmodified baseline. Identical SegResNet models were trained with Auto3DSeg/MONAI and evaluated against unmodified validation and test sets using the Dice similarity coefficient (DSC). Results: Mean test DSC decreased from 0.917 for the baseline model to 0.856 at 10 mm modification. Models trained with small annotation perturbations of 1–5 mm maintained DSC values of at least 0.90, whereas performance declined significantly beyond 5 mm. Pairwise DSC agreement across modified annotations also fell as modification amplitude increased. Conclusions: Prostate segmentation models tolerated modest annotation variability but degraded substantially when variability exceeded 5 mm, underscoring the importance of annotation quality when training and benchmarking DNN-based automated segmentation models.