What question did this study set out to answer?

This research investigates the effect of annotator experience on the performance of machine learning models for colorectal liver metastases characterization.

March 31, 2026Open Access

The Influence of Annotators' Experience on Radiomics-based Machine Learning Performance in Colorectal Liver Metastases Characterization: Impact and Mitigation Strategy

Key Points

This research investigates the effect of annotator experience on the performance of machine learning models for colorectal liver metastases characterization.
Analyzed an open-source CT dataset of 93 colorectal liver metastases annotated by four observers with varying expertise.
Developed a machine-learning model using nested cross-validation on a standardized radiomics pipeline that extracted 107 features.
Evaluated model performance under two conditions: standard normalization and observer-specific normalization.
The radiologist consistently outperformed other annotators, with the highest AUC-ROC scores in most scenarios.
The proposed observer fingerprint normalization improved model performance significantly across cross-annotator evaluations (p<0.001).
The findings indicate that annotator experience greatly influences model accuracy and generalization.

Abstract

Background: Segmentation variability is a major source of bias in radiomics, yet its quantitative impact on downstream model performance remains poorly defined.This study aimed to assess how annotator expertise influences model generalization and to test a mitigation strategy based on a observer fingerprint correction.Methods: An open-source CT dataset including 93 colorectal liver metastases with 46 desmoplastic and 47 replacement growth patterns was annotated independently by four observers with different expertise (radiologist, PhD student, two medical students).A standardized radiomics pipeline extracted 107 features and a machine-learning model was developed using nested cross-validation.Each model was evaluated in two setups: a standard pipeline with z-score normalization based on the training annotator, and a mitigation setup using external observer-specific normalization ("observer fingerprint").Statistical comparisons used Wilcoxon signed-rank tests with rank-biserial correlation and common language effect size.Results: In the first setup, the best performing test set annotator was the radiologist for both the models trained on radiologist, PhD and STUD1 annotations with an AUC-ROC median IQR of 0.79 0.76, 0.81, 0.74 0.70, 0.76, and 0.77 0.74, 0.80 respectively.For the model trained on STUD2 annotations, PhD was the best test set annotator with an AUC-ROC of 0.72 0.70, 074.In the second setup, the best performing test set annotator was RAD for all the models.The mitigation strategy significantly increased median AUC-ROC in the majority of cross-annotator comparisons (p<0.001).Conclusions: Model performance depends not only on the annotator used for training but also on the operator performing segmentation at deployment.The proposed mitigation strategy effectively reduced cross-annotator performance variability.

KI fragen

Bookmark

View Full Paper