December 1, 2025

Automatic Detection of Early Cognitive Decline Using Multimodal Feature Fusion and Transfer Learning on Real-World Conversational Speech

MPMadhurananda PaharUniversity of Sheffield BMBahman MirheidariUniversity of Sheffield CICaitlin Illingworth

Key Points

Cognitive decline detection achieves improved accuracy using multimodal feature fusion.
F1-scores of 0.83 and 0.54 were attained with a CNN/Bi-LSTM model for dementia and MCI classification.
Utilizing transfer learning led to a 3% performance increase while enhancing processing speed by 38%.
Findings support generalization across datasets as classifiers were effective on the DementiaBank dataset.

Abstract

Early signs of cognitive decline, such as dementia and mild cognitive impairment (MCI), often manifest in conversational speech. Early and accurate identification is essential for potential interventions prior to the onset of more severe stages of neurodegenerative diseases. We present CognoMemory, a system for detecting cognitive decline based on a person's speech, to collect 307 hrs of real-world conversational speech, corresponding to 1. 92 million Whisper-transcribed words, from 1, 639 participants. Speech recordings were collected as participants answered 14 memory-probing, clinically effective questions asked by a virtual agent, starting with a motivation prompt, followed by memory, cognitive functioning, fluency, picture description and reading task. Both acoustic and linguistic features, along with large language model (LLM) embeddings, were extracted from all 1, 639 participants. A subset of 614 participants, either with an unconfirmed diagnosis or younger than 50 years, was used for pre-training. The remaining three groups (64 dementia, 169 MCI and 792 healthy participants) were used to fine-tune our proposed model. Our multimodal feature fusion and CNN/Bi-LSTM-based transfer learning approach outperforms LLM-based (BART, DistilBERT, RoBERTa and HuBERT) approaches while achieving the highest F₁-scores of 0. 83 exhibiting a 3% performance increase due to the application of transfer learning, while being also 38% faster. Finally, the classifiers trained on the CognoMemory data, the largest of its kind, were tested on the second-largest available DementiaBank dataset (Pitt corpus), and a CNN-based transfer learning architecture achieved an F₁-score of 0. 89, demonstrating better stability and generalisation across datasets and of our novel feature fusion and architecture.

Ask AI

Helpful

Bookmark

View Full Paper