Panic prediction from digital phenotyping: subject-level cross-validation reveals limited between-person generalization

Objectives. A publicly shared digital phenotyping dataset was recently used to train models predicting day-before-panic (DBP) events, reporting cross-validated ROC-AUC around 0.90. Because the stratified 5-fold cross-validation allowed the same participant’s days in both training and test folds, it was unclear whether this reflects between-person generalization or information leakage. We reanalyzed the public dataset to quantify this distinction. Methods. We reanalyzed the public dataset (3,969 person-days; 254 day-before-panic events 6.4% across 19 of 43 participants) with the same XGBoost classifier and 71 features, comparing stratified 5-fold cross-validation with leave-one-subject-out (LOSO) cross-validation. A retrospective diagnostic decomposition applied within-subject z-scoring to dynamic features before LOSO. Results. We reproduced the original receiver operating characteristic area under the curve (ROC-AUC) at 0.895 ± 0.011. Under LOSO, pooled ROC-AUC dropped to 0.489 (bootstrap 95% CI: 0.325–0.643, including 0.50). The SHapley Additive exPlanations (SHAP) ranking under stratified CV was dominated by static traits (CTQ subscales, BRIAN, SPAQ). The diagnostic decomposition yielded LOSO ROC-AUC 0.516 (PR-AUC 0.114), suggesting weak residual within-person signal, though this should not be interpreted as prospective performance. Conclusions. Robust between-person generalization was not confirmed under subject-held-out evaluation. The feature ranking was trait-dominated, though the data cannot definitively distinguish identity discrimination from other between-person variance sources. Digital phenotyping studies should report subject-level cross-validation alongside standard metrics, per the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) and Prediction model Risk of Bias Assessment Tool (PROBAST) guidelines.