Deep convolutional neural networks information fusion and improved whale optimization algorithm based smart oral squamous cell carcinoma classification framework using histopathological images

Abstract The most prevalent type of cancer worldwide is mouth cancer. Around 2.5% of deaths are reported annually due to oral cancer in 2023. Early diagnosis of oral squamous cell carcinoma (OSCC), a prevalent oral cavity cancer, is essential for treating and recovering patients. A few computerized techniques exist but are focused on traditional machine learning methods, such as handcrafted features. In this work, we proposed a fully automated architecture based on Self‐Attention convolutional neural network and Residual Network information fusion and optimization. In the proposed framework, the augmentation process is performed on the training and testing samples, and then two developed deep models are trained. A self‐attention MobileNet‐V2 model is developed and trained using an augmented dataset. In parallel, a Self‐Attention DarkNet‐19 model is trained on the same dataset, whereas the hyperparameters have been initialized using the whale optimization algorithm (WOA). Features are extracted from the deeper layers of both models and fused using a canonical correlation analysis (CCA) approach. The CCA approach is further optimized using an improved WOA version named Quantum WOA that removes the irrelevant features and selects only important ones. The final selected features are classified using neural networks such as wide neural networks. The experimental process is performed on the augmented dataset that includes two sets: 100× and 400×. Using both sets, the proposed method obtained an accuracy of 98.7% and 96.3%. Comparison is conducted with a few state‐of‐the‐art (SOTA) techniques and shows a significant improvement in accuracy and precision rate.