Simultaneous multi-task assessment of five neurological language disorders via cross-corpus multimodal learning

Automated analysis of spontaneous speech may offer a scalable, objective complement to clinical assessment of neurological language disorders, yet existing approaches are largely corpus- and disease-specific. We propose a unified model that simultaneously classifies five neurological language disorders—Aphasia, Alzheimer’s Disease (AD), Mild Cognitive Impairment (MCI), Traumatic Brain Injury (TBI), and Right Hemisphere Damage (RHD)—and a healthy Control class across five heterogeneous TalkBank corpora (1,638 samples). To our knowledge, this is the first model to combine multimodal input (text, speech, and tabular metadata), multi-corpus training, and multi-task output for this many disorders, fusing modalities through a Q-Former cross-attention module that feeds a shared representation to a classifier and three auxiliary clinical heads (WAB-AQ severity regression, disease-similarity retrieval, and longitudinal progression). Under five-seed evaluation the model attains test accuracy \(90.98\%\) and Macro-F1 \(86.19\%\) , outperforming all baselines, and the auxiliary heads estimate aphasia severity within the WAB-AQ’s measurement error, retrieve same-disorder patients, and track recovery. Ablation identifies Q-Former fusion and class-frequency-weighted sampling as the principal performance drivers, and an inter-disorder correlation analysis explains the dominant MCI–Control confusion, recovers an etiology-based disorder grouping, and finds paralinguistic features among the top discriminators. Leave-One-Corpus-Out evaluation shows that zero-shot generalization to unseen disease classes remains fundamentally challenging.