Calibration failure and limited-label recalibration of retinal image classifiers under dataset shift

Medical image classifiers are often evaluated by discrimination metrics, although their probability estimates may fail on data from a different imaging source. This study evaluates probability calibration in one cross-dataset retinal image experiment and tests limited-label target recalibration without model retraining. Five EfficientNet-B0 classifiers were trained on APTOS 2019 and evaluated on a disjoint Messidor-2 test set. Internal APTOS AUC was 0.997 ± 0.002, whereas external Messidor-2 AUC was 0.665 ± 0.066. An audit found exact or confirmed near-identical images spanning the image-level APTOS splits; excluding the 27 affected source-test images left AUC essentially unchanged at 0.997 ± 0.002. Target calibration was repeated for 200 random samples at each size (25, 50, 100, and 200 labels) and compared across temperature, Platt, beta, and isotonic calibration. Temperature-scaling equal-mass ECE15 decreased from 0.108 at 25 labels to 0.085 at 200 labels; beta calibration reached 0.061 at 200 labels. At 200 labels, beta calibration achieved a mean Brier score of 0.221 and Brier skill score of 0.104 relative to a calibration-sample-prevalence predictor. The fixed 100-label temperature fit reduced high-confidence error rate (HCER) at 0.9 from 0.220 to 0.001, but coverage simultaneously decreased from 0.667 to 0.005. Risk–coverage curves were unchanged because temperature scaling preserves confidence ranking. Labeled target recalibration improved probability error but did not restore discrimination or remove classification errors. Confidence-threshold results therefore require coverage and conditional-error reporting.