When a Good Classifier Makes Bad Decisions: AUC as a Mann–Whitney Statistic, an Exact Variance Decomposition, and Cost-Optimal Thresholds Under Prior Shift in 41,188 Bank Marketing Calls

Banks increasingly choose whom to call, lend to or review using machine-learning scores, and they judge these scores mainly by the area under the ROC curve (AUC). I prove three results and test them on the UCI Bank Marketing data (41,188 telemarketing calls of a Portuguese bank, 2008–2010), using the first 80% of calls in time order for training and the last 20% for testing. First, AUC equals the Mann–Whitney probability that a random positive outscores a random negative, so it does not change under monotone transformations of the score or under changes in class prevalence. Second, its variance decomposes exactly into the DeLong terms plus a remainder bounded by 1/(4mn). Third, the cost-minimising decision calls a customer when the calibrated probability exceeds r/(1+r), where r is the cost of a wasted call relative to a missed sale, and at that point the ROC slope equals r(1−π)/π. Between the training and test periods the success rate rose from 6.37% to 30.83%. A logistic regression kept an AUC of 0.7483 (DeLong s.e. 0.0057), while gradient boosting, the better model on a random split (0.8150 against 0.8062), fell to 0.6358; the paired DeLong z-statistic is −20.06. Using the Bayes threshold with uncorrected probabilities raised costs above the best achievable threshold by 8.5–16.5% for the logistic model and by 43–182% for boosting. AUC alone hides these losses; the decision depends on calibration.