Gabor-VGG19 Model for Enhancing COVID-19 Detection through Multimodal Fusion using CT and CXR Images

Abstract Multimodal approaches have gained significant attention in medical research, combining information from different imaging modalities. This paper introduces a multimodal approach for the detection of COVID-19 using Computed Tomog-raphy (CT) and Chest X-ray (CXR) images. Employing a deep learning-based pre-trained VGG19 model, our focus is on evaluating the classification performance of the proposed modified VGG-19 in comparison to single-modal CT images, single-modal CXR images, a multimodal fusion of CT and CXR images, and a multimodal mixture of CT and CXR images. For experimentation, an imbalanced dataset is employed and investigate the impact of multimodality on such datasets. Three augmentation strategies—no augmentation, partial augmentation , and complete augmentation—are implemented to assess their effects on the algorithm’s performance and check robustness with the local dataset. The results indicate that the multimodality fusion-based approach with partial augmentation consistently outperforms both single-modalities and the mixture of multimodalities. This research contributes valuable insights into the potential of multimodal approaches in enhancing disease detection models, particularly in the context of imbalanced datasets associated with COVID-19 diagnosis.