Evaluating the Accuracy of Remote Questionnaire and Speech Measures in Detecting Cognitive Impairment in the ADNI4 Remote Digital Cohort: Cross-Sectional Study

Background: Scalable remote approaches are increasingly needed to identify cognitive impairment in clinical research. The Everyday Cognition (ECog) scale and its short form (12-item Everyday Cognition [ECog12]) capture subjective cognitive and functional decline, whereas automated speech-based story recall provides an objective measure of cognitive performance. Whether speech recall contributes information beyond remote questionnaire assessment remains unclear. Objective: This study aimed to determine whether automated speech-based story recall measures provide incremental discrimination of cognitive impairment diagnoses beyond the self-reported ECog12 (Self-ECog12) scale and demographics, both without and with 2 additional subjective memory measures that assess current memory/thinking concerns and perceived recent decline. Methods: This cross-sectional study included 7268 individuals from the Alzheimer's Disease Neuroimaging Initiative 4 (ADNI4) Remote Digital cohort with complete baseline questionnaire, demographic, and Storyteller speech-recall data. The outcome was defined as a participant report of a prior diagnosis of mild cognitive impairment or memory problem, Alzheimer disease, or dementia (n=574, 7.9%). Participants completed a smartphone-based Storyteller assessment involving immediate and delayed spoken recall of standardized stories; recordings were automatically analyzed to derive 5 prespecified recall and language measures. Participants were stratified into training and independent natural-prevalence test cohorts. Logistic regression models evaluated Self-ECog12 alone; Self-ECog12 plus demographics; Storyteller added to that model; the 2 additional subjective memory measures; and the full model incorporating Storyteller. Model development and threshold selection were confined to training data. Discrimination, calibration, and threshold-dependent performance were evaluated in the held-out test cohort. Area under the receiver operating characteristic curve (AUC) comparisons used paired DeLong tests. At the frozen threshold, the full model achieved sensitivity of 0.874 (95% CI 0.818-0.930), specificity of 0.746 (95% CI 0.725-0.767), positive predictive value of 0.227 (95% CI 0.210-0.246), and negative predictive value of 0.986 (95% CI 0.980-0.992). The Brier score was 0.060, with a calibration intercept of 0.023 and a slope of 0.870, suggesting acceptable calibration with overfitting. Conclusions: Speech measures provided incremental information beyond Self-ECog12 and demographics, with and without additional subjective memory measures. These findings support further evaluation of multimodal remote assessment as a scalable screening or enrichment approach for identifying individuals who may benefit from clinical evaluation. Because the outcome was a participant-reported prior diagnosis rather than an independently adjudicated current diagnosis, validation against clinically adjudicated outcomes is necessary before diagnostic use.