A novel hybrid framework integrating GA-driven 3D ResUNetGAN for MRI brain tumor segmentation

Accurate brain tumor segmentation by multi-modal MRI is crucial for diagnosis, treatment planning, and prognostic assessment. This work proposes a novel hybrid architecture within a GA optimized by 3D residual U-Net (ResUNet) with generative adversarial network (GAN) for the accurate segmentation of tumor subregions, namely the tumor core (TC), whole tumor (WT), and enhancing tumor (ET), using a dataset collected from BraTS2023 (437 training and 188 validation). By using GAs, we enable the computer to automatically find the best model structure and training settings, which helps keep the model simple enough to avoid overfitting while being powerful enough to generalise well to new data. The GAN part of our method trains the model to draw more realistic and consistent boundaries around tumor regions in MRI scans, thereby improving the overall quality of the segmentation. In our experiments, this GA-ResUNetGAN approach performed better than several leading methods, such as 3D UNet, DynUNet, UNETR, SwinUNETR, and the standard ResUNet, especially in accurately identifying TC, WT, and ET parts—achieving Dice scores of 0.94, 0.96, and 0.91 on the validation set. In addition, K-Fold cross-validation was performed to ensure consistent performance across different data splits. While the method involves more processing work than some baseline techniques, it provides improved segmentation quality and boundary delineation on the evaluated dataset. These results demonstrate the advantages of combining evolutionary optimization with adversarial deep learning for challenging medical imaging tasks.