Development of a Novel Machine Learning Model for Automatic Assessment of Quality of Transvaginal Ultrasound Images From Multi‐Annotator Labels

Objectives Accurate diagnosis of pathology from ultrasound images is reliant upon images of a suitable diagnostic quality being acquired. This study aimed to create a novel machine learning model to automatically assess transvaginal ultrasound (TVUS) image quality for gynaecological ultrasound. Method Six imaging professionals (two sonographers, two gynaecological sonologists and two radiologists) assigned a quality score to 150 TVUS images from 50 cases (50 uterus images and 100 ovary images). Images were given a score of 1–4 (1—reject/image inaccurate, 2—poor quality, 3—suboptimal quality or 4—optimal quality). As variation existed between the scores assigned by the labellers, we framed this problem as a multi‐annotator noisy label problem. To address this, a new machine learning architecture was developed, combining a weighted ensemble algorithm to estimate consensus labels and a multi‐axis vision transformer (MaxViT) to handle noisy labels, improving model accuracy in predicting image quality. Forty cases (120 images) were used for model training, while the remaining 10 cases (30 images) were reserved as a test set for model evaluation. Results The novel machine learning architecture we created was able to successfully determine image quality with a validation accuracy of 80% and a macro average recall of 77%. This significantly improved upon the 57% accuracy of the baseline machine learning method (ResNet50). The MaxViTs were able to outperform human performance in most cases, with an accuracy of 80% surpassing four of the six human labellers. Conclusions This novel machine learning model offers an automated method of assessing the quality of TVUS images. The tool has the potential to provide real‐time feedback to those performing TVUS, reduce the need for repeated imaging, and improve the diagnosis of gynaecological pathology.