When AI and Experts Agree on Error: Intrinsic Ambiguity in Dermatoscopic Images

The integration of artificial intelligence (AI), particularly convolutional neural networks (CNNs), into dermatological diagnosis demonstrates substantial clinical potential. While the existing literature predominantly benchmarks algorithmic performance against human experts, our study adopts a novel perspective by investigating the intrinsic complexity of dermatoscopic images. Through rigorous experimentation with multiple CNN architectures, we isolated a subset of images systematically misclassified across all models—a phenomenon statistically proven to exceed random chance. To determine whether these failures stem from algorithmic biases or inherent visual ambiguity, expert dermatologists independently evaluated these challenging cases alongside a control group. The results revealed a collapse in human diagnostic performance on the AI-misclassified images. First, agreement with ground-truth labels plummeted, with Cohen’s kappa dropping to a mere 0.08 for this subset, compared to 0.61 for the control group. Second, we observed a severe deterioration in expert consensus; inter-rater reliability among physicians fell from moderate concordance (Fleiss’ kappa = 0.456) on control images to only modest agreement (Fleiss’ kappa = 0.275) on the misclassified subset. We identified image quality as a primary driver of these dual systematic failures. To promote transparency and reproducibility, all data, code, and trained models have been made publicly available.