Improving Heart Disease Classification using Multimodal Fusion in Deep Learning

Heart disease is one of the causes of suffering for those affected and is also one of the leading causes of death. There are many studies that examine the diagnosis of diseases, and deep learning has emerged as a prominent tool in their diagnosis and differentiation. Previous studies relied on classifying heart diseases using deep learning techniques, either using clinical data or relying on electrocardiogram signals. This study focused on using the Multimodal Fusion in Deep Learning Neural Network MFDNN model to diagnose lung cancer through various types of data sets (imaging, genomic, clinical). In this research, a three-steps model was created. The first step extracts feature from images using the ResNet18 network, the second step extract features from clinical data by uses a neural network containing linear layers , and the final step combines the features extracted from the first and second steps and classifies them. The MFDNN model achieved a training accuracy of 99.69% and a testing accuracy of 99.93%. The proposed model also excelled in classifying each category of heart disease perfectly, with F1-score, Recall, and Precision values reaching 100% for each category. The proposed model was compared with previous studies and found to be superior in terms of reliance on clinical data as well as electrocardiography ECG tracings.