Penalized Logistic Versus Additive Spline Models for Predicting Academic Failure: A Repeated Nested Cross-Validation Study of Discrimination and Calibration

Statistical models that estimate a student’s risk of an unsuccessful academic outcome are increasingly proposed for allocating academic support, but such models must both rank students accurately and produce well-calibrated probabilities. This study compared ridge-penalized logistic regression with additive logistic models built from cubic B-spline bases for predicting academic failure, defined as failure or withdrawal. Five nested specifications added background, engagement and assessment predictors in sequence and were evaluated by stratified five-fold cross-validation repeated three times, with the penalty tuned in an inner loop. Performance was measured by the area under the receiver operating characteristic curve (AUC), average precision, Brier score, log loss and calibration intercept and slope. Background variables alone gave no improvement in Brier score over a constant-risk prediction (AUC 0.543; Brier score 0.1254). Engagement variables added little (AUC 0.594; Brier score 0.1240), whereas assessment variables produced the largest gain: the assessment-inclusive logistic model achieved an AUC of 0.736, average precision 0.318, Brier score 0.1144 and calibration slope 0.942. Its spline counterpart did not improve on it (AUC 0.721; Brier score 0.1154; slope 0.904). With 147 unsuccessful outcomes, added flexibility did not improve internal predictive performance, favouring the parsimonious logistic model. The results are internal to one district survey and require external and temporal validation before any operational use.