Probability Calibration Under Class-Prevalence Shift: A Reproducible Multi-Dataset Stress Test

Probability estimates are routinely transported from model-development samples to populations with different outcome prevalence, yet discrimination can remain apparently strong while predicted risks become unreliable. This study quantifies that failure and evaluates whether post-hoc calibration and analytically specified prior-probability correction restore probabilistic accuracy. Three public classification datasets spanning clinical, chemical, and image-recognition tasks were evaluated with logistic regression, random forests, and histogram gradient boosting. Each dataset underwent 20 stratified repetitions with separate training, calibration, and test pools. Platt scaling and isotonic regression were compared with uncalibrated probabilities. Test sets of 400 observations were resampled at positive-class prevalences of 0.10, 0.25, 0.50, 0.75, and 0.90, producing 5,400 model–calibration–shift evaluations. Outcomes included area under the receiver-operating-characteristic curve (AUROC), Brier score, logarithmic loss, expected calibration error (ECE), and balanced accuracy. Across uncalibrated models, prior correction reduced mean Brier score from 0.0404 to 0.0316 (21.7%) and improved Brier score in 692 of 900 paired comparisons. Platt scaling with prior correction achieved the lowest overall Brier score (0.0297); isotonic regression with correction achieved the lowest ECE (0.0273). AUROC was unchanged by correction, demonstrating that ranking performance does not diagnose probability transportability. However, balanced accuracy at a fixed 0.5 threshold declined under extreme prevalences, showing that probability correction and decision-threshold selection are distinct tasks. Probability models should therefore be audited under plausible prevalence shifts and evaluated with proper scoring rules, calibration measures, and application-specific decision thresholds rather than discrimination alone.