A new hybrid brain tumor detection and segmentation with enhanced ResNet-50 and attention-based residual U-Net architecture

Brain tumor (BT) detection and segmentation from magnetic resonance imaging (MRI) are critical for accurate diagnosis and treatment planning. MRI is a valuable tool for tumor localization; however, existing approaches often struggle with complex tumor structures, class imbalance, and high computational costs. Currently, advanced methods for BT detection and segmentation rely on both traditional machine learning (ML) models and fully convolutional neural networks (FCNs) employing either 2D or 3D convolutions. Conventional ML models often face challenges in capturing the nonlinear relationships inherent in medical MRI brain images. Accurate BT identification and segmentation require the analysis of complex spatial patterns, which such models may fail to adequately represent. Furthermore, 2D convolutions cannot fully exploit the spatial information in volumetric brain images, while 3D convolutions incur substantial computational and memory overhead. Motivated by these challenges, we propose a new hybrid two-stage deep learning framework for automated brain tumor analysis. The first stage employs a pretrained ResNet-50 model for tumor detection, while the second stage utilizes an enhanced Attention Residual U-Net with focal Tversky loss (FTL) for precise segmentation. In addition, a hybrid feature optimization strategy combining recursive feature elimination (RFE) and principal component analysis (PCA) is introduced to improve feature selection and representation. The FTL is further applied to enhance pixel-level segmentation performance. Experimental evaluation on publicly available MRI datasets demonstrates superior performance, achieving an accuracy of 99.80%, recall of 99.13%, precision of 99.20%, and an F1 score of 99.0%. The proposed Attention Res-UNet-FTL model outperforms state-of-the-art models, including MobileNet, VGGNet, QuickNAT, XceptionNet, and DenseNet. It achieves a Dice score of 84.77% and an intersection-over-union (IoU) of 99.18%. The proposed approach provides a reliable and efficient solution for BT analysis, with strong potential for integration into clinical decision-support systems.