Accurate brain tumor classification from neuroimaging modalities such as Magnetic Resonance Imaging (MRI) and Computed Tomography (CT) is critical for clinical diagnosis. However, fine-tuning large deep learning models on moderately sized medical datasets often demands extensive computational resources. This work demonstrates that strategic transfer learning can address this challenge. We propose a dual-branch architecture that combines two pre-trained backbones: a ConvNeXt to capture local patterns and a Swin Transformer to model global contextual dependencies. A cross-attention mechanism fuses these complementary features, while a differential learning rate strategy accelerates convergence. Extensive evaluation on eight publicly available brain MRI and CT datasets spanning binary and multi-class classification tasks shows that our model outperforms state-of-the-art baselines on most datasets and metrics and matches the best-performing models in remaining cases, all with substantially fewer training epochs. The main contribution of this work is the introduction of a dual-branch fusion architecture coupled with a differential learning rate strategy, enabling efficient adaptation of pre-trained backbones for medical image analysis.
Esfahani et al. (Sat,) studied this question.