Lesion-Aware Dual Attention Fusion with MedSAM-2 and Hierarchical Swin-SENet for Robust Diabetic Retinopathy Grading

Diabetic retinopathy (DR) ranks as the major cause of preventable blindness in the world, and is estimated to affect some 160 million diabetics. Severity grading demands a very accurate identification of subtle, spatially heterogeneous, retinal lesions in 5 clinical stages. Current deep learning pipelines regard lesion segmentation and disease classification as separate processes and do not make use of spatial pathological priors for feature learning. To this end, this study aims to develop a unified segmentation-guided dual-attention framework to explicitly fill this methodological gap by designing a learnable lesion-aware attention fusion mechanism. Medical Segment Anything Model-2 (MedSAM-2; ViT-B backbone) to provide a soft lesion probability map on fundus images. This map is then translated to learnable spatial attention weights by using a novel Lesion-Aware Attention Fusion (LAAF) module, which reweights intermediate features of a hierarchical Swin-SENet classifier by boosting pathological regions while maintaining the global retinal context. Inter-class discriminability is further improved through channel-wise recalibration through Squeeze-and-Excitation (SE) blocks. The framework is trained end-to-end using a segmentation and weighted cross-entropy loss. MedSAM-2 achieves a segmentation IoU of 0.8978 and Dice coefficient of 0.8843 on IDRiD (988 augmented images). Five-class DR grading achieves 92.7% accuracy, F1-score of 0.92 and Cohen's Kappa of 0.89, with bootstrap 95% CI of 91.1-94.2%. The LAAF module shows the highest improvement for Mild and Moderate DR (F1: 0.88, 0.90). It is cross-dataset generalizable with 89.3% accuracy and kappa of 0.85 when tested on APTOS 2019 (3,662 images). The LAAF module offers a principled learnable route to incorporate segmentation priors into transformer-based grading networks, which is shown to enhance lesion sensitivity, inter-grade separability and diagnostic reliability on various benchmark datasets.