SCANFormer: scale and context-aware nested feature fusion transformer for medical image segmentation

Accurate delineation of target organs in medical images remains challenging due to ambiguous boundaries. Although current approaches for medical image segmentation demonstrate strong performance, they come with certain limitations. Convolutional Neural Network (CNN)-based models are incapable of capturing long-range dependencies; Transformer-based models often overlook multi-scale feature interactions; and existing hybrid models lack effective multi-scale feature fusion and generalizability. We propose Scale and Context-Aware Nested Feature Fusion Transformer (SCANFormer), a generalized framework for medical image segmentation. SCANFormer introduces a novel, comprehensive, multi-scale feature extraction and fusion approach to enable more diverse feature learning. The encoder captures spatial features at multiple scales and combines them using the Multi-Head Convolution (MHC) and Scale-aware Feature Aggregation (SFA) modules. These representations are further enhanced by Scale-aware Feature Refinement (SFR). Semantic alignment across encoder and decoder stages is further improved with redesigned nested skip connections enhanced by Efficient Attention (EA). Extensive experiments on four computed tomography (CT) and magnetic resonance imaging (MRI) datasets, including ACDC, Synapse, LCTSC, and BraTS-Africa2024, demonstrate the superiority of SCANFormer, achieving 3%, 8.48%, 11.60%, and 11.06% higher Dice scores than TransUNet, respectively, even when trained from scratch. These results highlight SCANFormer’s ability to accurately segment the organ boundaries and generalize across modalities and anatomical structures. Pretrained weights, source code and sample prediction results for all four datasets are deployed at: https://github.com/Sri-Kavya-Ketagani/SCANFormer .