Auxiliary Classifier GAN-based Synthetic Histopathological Image Generation with Deep Ensemble Pipeline for Enhanced Oral Squamous Cell Carcinoma Detection

Background: Oral squamous cell carcinoma (OSCC) remains a challenging issue in cancer treatment, owing to the lack of accurate diagnosis models, resulting in delayed detection and treatment. Histopathological assessment of tissue specimens is fundamental for the detection of OSCC. The occurrence of understated visual variations among early malignant and benign oral conditions leads to misclassification, affecting early identification and survival rates. Objectives: Therefore, this study develops a new Generative Adversarial Network-based Synthetic Image Generation with Deep Ensemble Pipeline (GANSIH-DEP) technique for enhanced OSCC detection and classification. The aim of the GANSIH-DEP technique is to balance the OSCC dataset via data augmentation and auxiliary classifier generative adversarial network (AC-GAN) based synthetic image generation, thereby improving the Deep Learning (DL) model performance on OSCC detection. Methods: Data augmentation takes place via geometric ways (rotation, flipping, scaling, translation) to raise data diversity during training with the same class labels. Moreover, the GANSIH-DEP technique utilizes AC-GAN to generate high-fidelity synthetic images to resolve the limited data availability and class imbalance problem, specifically enhancing the representation of rare and atypical OSCC classes, which are often underdiagnosed in small clinical datasets. Furthermore, the deep feature representations are extracted using a multi-branch convolutional neural network (CNN)-based model. To optimize the performance of a multi-branch Convolutional neural network (CNN) model, the Nadam optimizer is used for hyperparameter selection to accomplish stability and high convergence. For OSCC detection, the weighted deep learning ensemble (WDLE) model is used to enhance diagnostic performance by minimizing model bias and variance. Final, the Explainable Artificial Intelligence (XAI) using the Grad–Class Activation Maps (CAM) is integrated to provide visual explanations by highlighting the discriminative regions influencing the classification outcomes. Results: The experimental validation of the proposed model on the benchmark OSCC dataset exhibits an improved accuracy of 98.07%, demonstrating reduced false negatives, assuring that the early-stage lesions are not unnoticed. The enhanced performance significantly supports earlier clinical decision making, enhanced treatment planning, better prognosis, and enhanced long-term survival rate. Conclusion: The proposed model is found to be an efficient diagnostic tool for the robust OSCC detection process.