Deep learning to predict future cognitive decline: a multimodal approach using brain MRI and clinical data

Predicting the trajectory of clinical decline in aging individuals is a pressing challenge, especially for people with mild cognitive impairment, Alzheimer's disease, Parkinson's disease, or vascular dementia. Accurate predictions can guide treatment decisions, identify risk factors, and optimize clinical trials. In this study, we compared two deep learning approaches for forecasting changes, over a 2-year interval, in the Clinical Dementia Rating scale 'sum of boxes' score (sobCDR), as a continuous outcome (regression). This is a key metric in dementia research and clinical trials, and scores range from 0 (no impairment) to 18 (severe impairment). To predict decline, we trained a hybrid convolutional neural network (CNN) that integrates 3D T1-weighted brain MRI scans with tabular clinical and demographic features (including age, sex, body mass index (BMI), and baseline sobCDR). We benchmarked its performance against AutoGluon, an automated multimodal machine learning framework that selects an appropriate neural network architecture (an 'autoML' approach). We evaluated the models using data from 2,319 unique participants drawn from three independent cohorts-ADNI, OASIS-3, and NACC. For each participant, we used one T1-weighted brain MRI scan along with corresponding clinical and demographic information. Our results demonstrate the importance of combining image and tabular data in predictive modeling for this clinical application. Deep learning algorithms can fuse information from image-based brain signatures and tabular clinical data, with potential for personalized prognostics in aging and dementia. Rather than concluding that multimodal fusion uniformly improves performance, our results show that deep learning applied to volumetric MRI data may struggle to add predictive value, particularly when clinical covariates explain substantial variance and provide a strong baseline. In other conditions and tasks, it may help to have a hybrid system that can learn from both data types, and their relative value may be different. Conversely, AutoML-based multimodal fusion provides a robust baseline when tabular data already provide strong predictive value for the task. These insights clarify how different multimodal strategies could be selected in clinical prognostic applications.