Reliable and calibrated diabetic retinopathy grading from Gaussian-filtered fundus images using attention and gated state-space feature mixing

Diabetic retinopathy (DR) is a leading cause of preventable vision loss, and automated severity grading is essential for scalable screening; however, existing deep-learning approaches are limited by inadequate long-range contextual modeling, severe class imbalance, and poorly calibrated predictions. To address these challenges, we propose ABSS-Net, a compact (5.47 M parameters) Attention-Based State Space Network that integrates an ImageNet–Noisy-Student pretrained EfficientNet-B0 multi-scale feature pyramid with hierarchical Convolutional Block Attention Modules (CBAM) and stacked gated State Space (SSM) blocks for efficient global feature mixing. A dual classification–reliability head is optimized using a hybrid objective combining class-weighted focal loss ( \(γ = 2.0\) ) and a Brier-style calibration penalty ( \(λ = 0.1\) ). Two experimental protocols on the Diabetic Retinopathy \(224× 224\) Gaussian-filtered dataset (3,662 images, five ICDR classes) are reported. Under an 80/20 stratified development and test split, ABSS-Net achieves an accuracy of \(76.40%\) , macro-F1 of 0.6107, QWK of 0.8563, AUC of 0.9235, and ECE of 0.0507. Additionally, statistical stability is evaluated using stratified 5-fold cross-validation, yielding \(70.26 ± 4.20%\) accuracy, \(0.5206 ± 0.0553\) macro-F1, \(0.7933 ± 0.0418\) QWK, \(0.8862 ± 0.0168\) AUC, and \(0.1778 ± 0.0439\) ECE, demonstrating that the architecture remains statistically stable across partitions. The self-reflective reliability head independently achieves high accuracy for high-confidence predictions ( \(c ≥ 0.76\) ; \(94.27%\) ) and lower accuracy for low-confidence predictions ( \(46.26%\) ). A prediction–rejection analysis (AURC \(= 0.1050\) ) quantitatively supports a clinical-deferral workflow at \(81.00%\) accuracy and \(70%\) coverage. A Messidor-2 zero-shot external evaluation reveals a substantial domain shift, which is analyzed and identified as a direction for future work. The contribution of each architectural component and loss term is validated through ablation studies. ABSS-Net, leveraging local attention, long-range state-space modeling, and explicit calibration learning, delivers accurate, reliable, and explainable DR grading. This is further supported by Grad-CAM visualizations that align with clinically relevant retinal regions, supporting further investigation towards tele-ophthalmology screening, contingent on the external-domain validation discussed below.