Reliable or Just Accurate? A Cross-Dataset Audit of Early-Warning Models Under Course-Level Distribution Shift

Early-warning systems based on Learning Management System (LMS) traces are commonly evaluated using random or unseen-student splits, even when deployment requires predictions in courses that were absent during model development. In this study, we distinguish predictive accuracy within familiar course environments from reliability under course-level distribution shift. We evaluate an institutional Moodle deployment from the International Hellenic University (IHU; 1284 student-course observations, 924 students, 35 courses) alongside two public behavioral benchmarks, OULAD (N = 22,437) and Riestra (N = 25,260). Six semantically harmonized behavioral features were measured at 10%, 25%, 33%, and 50% observation cutoffs and evaluated under random-row, unseen-student, and unseen-course validation. With Extra Trees, unseen-course balanced accuracy was stable and above chance for OULAD (0.619–0.686) and Riestra (0.584–0.639), whereas IHU performance ranged from 0.470 to 0.582 and was statistically indistinguishable from chance at the earliest cutoff. The early IHU result persisted across Logistic Regression, Random Forest, Extra Trees, and Histogram Gradient Boosting. A grade-scale audit identified substantial inconsistencies between configured and observed course maxima; an observed-maximum sensitivity specification materially changed class prevalence and prevalence-sensitive metrics but did not significantly improve unseen-course balanced accuracy or ROC-AUC. Repeated size-matched controls attenuated the Riestra–IHU difference but left the OULAD advantage largely unchanged. OULAD leave-one-presentation-out and IHU leave-one-mixed-course-out analyses were consistent with grouped five-fold validation. The findings show that high within-sample accuracy is not sufficient evidence of operational reliability. Reported performance is associated with validation design, label definition, course composition, and dataset scale. We recommend unseen-course evaluation, label-definition audits, course-aware uncertainty estimates, and multi-model robustness checks as standard reporting practices for LMS early-warning systems intended for use beyond their original training courses.