NeuroFusionNet: an integrated Swin Transformer and CNN–RNN framework

This paper proposes a novel multimodal deep learning architecture to address the challenging task of brain tumor segmentation and classification from MRI images. The model combines the strengths of Swin Transformer, Recurrent Neural Networks (RNNs), and Convolutional Neural Networks (CNNs) to achieve strong segmentation and classification performance. Using the three-dimensional and sequential features of MRI images, the model provides comprehensive and precise brain tumor analysis. The framework consists of two primary segments: Swin Transformer and CNN-RNN. The Swin Transformer module extracts spatial features from MRI images by using a hierarchical approach to capture multi-scale features. This module provides a strong representation of tumor spatial features, which is crucial for precise segmentation and classification. The CNN-RNN module captures sequential relationships present in MRI images. The module leverages the CNN-RNN integration to capture progressive correlations between images. By combining the spatial features extracted by the Swin Transformer module with the sequential features captured by the CNN-RNN module, the framework provides a complete representation of brain tumors. The proposed method achieved an accuracy of 85.6% compared with other existing deep learning models. The proposed model also achieved an F1 score of 85%, which is also superior to other existing models. It uses the BraTS 2023 dataset for multimodal brain tumor segmentation and classification, with pixel intensity values from 0 to 255, and employs an 80/10/10 training-validation-testing split. The framework’s ability to accurately segment and classify brain tumors in clinical MRI images could help clinicians diagnose and treat brain tumors more effectively. Medical practitioners can use our proposed approach to quickly correlate patients’ MRI details to analyse tumor presence. Moreover, the framework’s modular design allows easy implementation across other medical imaging tasks, making it a valuable resource for evaluating medical images.