Explainable Deep-Learning-Based Diagnosis of Alzheimer’s Disease Using Multimodal Input Fusion of PET and MRI Images

Abstract Purpose Alzheimer’s disease (AD) is a progressive, incurable human brain illness that impairs reasoning and retention as well as recall. Detecting AD in its preliminary stages before clinical manifestations is crucial for timely treatment. Magnetic Resonance Imaging (MRI) provides valuable insights into brain abnormalities by measuring the decrease in brain volume expressly in the mesial temporal cortex and other regions of the brain, while Positron Emission Tomography (PET) measures the decrease of glucose concentration in the temporoparietal association cortex. When these data are combined, the performance of AD diagnostic methods could be improved. However, these data are heterogeneous and there is a need for an effective model that will harness the information from both data for the accurate prediction of AD. Methods To this end, we present a novel heuristic early feature fusion framework that performs the concatenation of PET and MRI images, while a modified Resnet18 deep learning architecture is trained simultaneously on the two datasets. The innovative 3-in-channel approach is used to learn the most descriptive features of fused PET and MRI images for effective binary classification of AD. Results The experimental results show that the proposed model achieved a classification accuracy of 73.90% on the ADNI database. Then, we provide an Explainable Artificial Intelligence (XAI) model, allowing us to explain the results. Conclusion Our proposed model could learn latent representations of multimodal data even in the presence of heterogeneity data; hence, the proposed model partially solved the issue with the heterogeneity of the MRI and PET data.