Instability and Overinterpretation of Predictive Performance in a Small Multimodal Clinical Dataset: A Methodological Case Study of Adult ADHD Versus Other Psychiatric Disorders

Small public multimodal datasets invite overstated claims that wearable signals classify attention-deficit/hyperactivity disorder (ADHD). The HYPERAKTIV dataset served as a methodological case study: 103 adult psychiatric outpatients, 51 with ADHD and 52 clinical controls, with actigraphy, heart-rate recordings, and cognitive testing. The task was differential classification, not screening. Background/Objectives: The objective is to quantify how far the reported performance depends on the evaluation protocol. Methods: The methods evaluated four nested feature blocks on 63 complete cases with one fixed model and one estimand: the mean out-of-fold area under the receiver operating characteristic curve (AUROC) over 20 fold assignments. Each block was compared with a permutation null from the identical protocol. Results: The results are that the fold assignment alone moved a single block’s AUROC by up to 0.235, and no block exceeded its null. The heart-rate-variability increment was −0.039 (percentile interval −0.153 to 0.155), inconclusive under every interval method. A bootstrap holding the fitted models fixed suggested a bounded null, and DeLong’s test made the cognitive increment significant; resampling the whole pipeline confirmed neither. On datasets with the heart-rate-variability block permuted, these two tests rejected in 18.3% and 14.7% of those datasets at a 5% level. The pre-scored ADHD Confidence Index alone reached an AUROC of 0.743, descriptively above every fitted model. Conclusions: The conclusions drawn are that small-sample studies should report a protocol-matched null and intervals that refit the whole pipeline.