Ensemble models for diabetic retinopathy detection and classification using vision transformers and capsule networks with advanced feature extraction techniques

Diabetic Retinopathy (DR) is a major complication of diabetes, leading to retinal degeneration and potential vision loss. Early and accurate detection is critical for timely intervention. This study proposes an advanced Ensemble Deep Learning model to detect and classify DR into five severity stages using fundus images. The model integrates a Transformer Neural Network (TNN) combining Vision Transformer and Recurrent Transformer architectures with Capsule Neural Network for robust classification. A comprehensive preprocessing pipeline enhances contrast and reduces noise to improve image quality. Feature extraction utilizes Gabor filters, Histograms of Oriented Gradients (HOG), and Local Binary Patterns (LBP) to capture texture, surface, and structural details. Dimensionality reduction is performed using Principal Component Analysis (PCA) and Recursive Feature Elimination (RFE), followed by optimal feature selection through a novel Hybrid Manta Rays-based Zebra Optimization Algorithm (HMZOA). The combined TNN-Capsule model effectively classifies DR with high accuracy. On the Eyepacs dataset, it achieved 98.92% accuracy, 98.91% specificity, and an F1-score of 99.02%. Comparable performance was recorded on the Aptos (98.58%) and Idrid (98.29%) datasets. These results demonstrate the model's high precision and reliability, contributing significantly to early DR detection and treatment planning.