Advancing Alzheimer’s Disease Detection via Multimodal Fusion of Speech and Eye Movement Data

This is the authors' abstract. We don't add key points for this paper.

Alzheimer's disease (AD) is a neurodegenerative disorder that impacts multiple cognitive domains. Early-stage changes in eye movement patterns and language abilities are often detectable and become more pronounced as cognitive decline progresses. This study investigates whether integrating speech and extraocular movement (EOM) data improves AD detection compared to using each modality alone, assessing its effectiveness across various modeling frameworks and language tasks. Our contributions include: (1) developing task-agnostic methods that leverage raw speech and EOM signals to improve the scalability and robustness of AD classification; (2) introducing SpeechEyeNet, a transformer-based cross-modal network that integrates speech and EOM data for effective multimodal fusion; (3) evaluating a pretrained time-series model for encoding raw EOM data, using its zero-shot capabilities for eye movement analysis without task-specific fine-tuning; and (4) conducting a comparative evaluation of interpretable and non-interpretable multimodal frameworks, highlighting their trade-offs and complementary strengths. Our findings indicate that multimodal fusion consistently outperforms unimodal baselines, with interpretable methods typically yielding superior performance, likely due to sample size limitations. This study underscores the complementary role of speech and EOM data in AD detection. Future work will consider different neurological disorders, explore new fusion techniques, and incorporate diverse datasets to improve model generalizability.Clinical relevance-This study highlights the potential of speech and EOM signal integration for improving early and accurate AD detection. Leveraging complementary modalities, our approach enhances diagnostic robustness and supports the development of non-invasive screening tools for clinical and real-world applications.