Key result
AI trained on naive annotations achieves similar LAA ultrasound segmentation accuracy to expert-trained models.
Why the study?
Supervised AI segmentation performance depends on operator expertise for ground truth generation, especially in challenging imaging modalities like ultrasound, but the impact of user expertise remains to be investigated.
Does the level of annotation expertise affect the performance of AI-based ultrasound segmentation of the left atrial appendage?
Does the level of annotation expertise affect the performance of AI-based ultrasound segmentation of the left atrial appendage?
AI-based ultrasound segmentation of the LAA is robust to non-systematic annotation errors mimicking inter-observer variability, but struggles with systematic errors.
May support novice LAA annotations for AI training; leaves open whether systematic error correction aids clinical use.
BACKGROUND: Medical image segmentation using artificial intelligence (AI) is a prominent area of research with diverse applications across various fields. During the last years, a multitude of datasets representing different body structures have been developed and made publicly available. However, the volume of data-particularly the ground truth data, which often relies on manual annotation-remains limited. Supervised learning remains the state-of-the-art approach for deep learning methods; however, its performance is often reported as dependent on the expertise of the operator for the ground truth generation. This dependency becomes more critical when dealing with challenging medical imaging modalities, such as ultrasound, often characterized by low image quality and various artifacts. METHODS: This study aims to investigate the influence of user expertise on the accuracy of ground truth annotations and their impact on the final performance of the segmentation method. Specifically, we focus on the task of segmenting the left atrial appendage (LAA) in ultrasound images. Two datasets were initially created: one annotated by an Expert and the other by a novice observer. Additionally, synthetic variations of these manually annotated datasets were generated by introducing both systematic and non-systematic errors to examine their effects on segmentation outcomes. RESULTS: Using the nnU-Net framework as the computational basis, the network was trained on each dataset, and the results were evaluated against the Expert's test labels. Training with Expert and Naive contours achieved Dice values in the test set of 0.81 ± 0.09 and 0.77 ± 0.12, respectively, with no statistically significant differences between them. Similarly, training with synthetic variations obtained showed no statistically significant differences for non-systematic errors, whereas systematic errors result in statistically significant differences against manual contours. CONCLUSIONS: These findings demonstrate that the AI network remains highly effective across most tested scenarios, even when synthetic errors are introduced, showcasing its ability to handle non-systematic errors efficiently, which synthetically mimic the variability between observers. However, the network encounters greater challenges with systematic errors, failing to accurately delineate the LAA boundaries.
No takes yet. Share an insight, caveat, or question.
Fernandes et al. (2026) studied Left atrial appendage segmentation. Expert annotations vs. Naive annotations was evaluated on Dice values in the test set. AI-based ultrasound segmentation of the left atrial appendage trained on expert versus naive annotations achieved similar Dice scores (0.81 vs 0.77), though systematic errors impaired performance.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: