Large-Scale Evaluation of the Influence of AI-use on Radiologist Performance using Signal Detection Theory

Research on the diagnostic performance of radiologists using AI decision support systemoften relies on empirical ROC-AUC and similar performance metrics to draw conclusions aboutclinical decision-making, with known limitations. We re-evaluated a real-world dataset ofradiologist use of AI across 127 pathologies and 1,163 cases using Signal Detection Theory, amodel of decision-making from cognitive science to better understand the impact of AI onclinical decisions. Our results revealed important insights; with increased disease detection(higher sensitivity) at the cost of more false positives (lower specificity) and increasedhomogeneity of decision making amongst clinicians whose diagnoses are typically highlyvaried. Although usually considered a positive property, we warn that homogenisation mayhave negative implications at a system level by homogenising incorrect diagnoses. Further,we find radiologists may be surprisingly robust to AI false negatives, leading to implicationsfor further research. We conclude that the effects of introducing AI in clinical practice areoutside of the scale of more common changes to practice, i.e. staffing, and that introducingclinical AI may have unpredictable consequences for human decision-making.