An Enhanced Convolution Transformer With Ladder Self‐Attention for Breast Cancer Histopathological Image Classification and Recurrence Prediction

The most common cancer that kills women worldwide is breast cancer. Early detection of breast cancer and the prediction of recurrence is essential for diagnosis because metastasis and recurrence account for the majority of cancer‐related deaths. Images from the BreakHis dataset are gathered because physicians frequently use histopathological images to classify breast cancer. It takes a pathologist and specialists to grade histopathological images, and the process is inaccurate and time‐consuming. Therefore, using a score‐based VGG‐16 model, an improved convolution transformer with ladder self‐attention was developed for the purpose of classifying breast cancer histopathological images and predicting recurrence. Initially, the images are pre‐processed using a color deconvolution filter. Convolution transformers with Ladder Self‐Attention units and attention‐aware frequency domain transformers are used to extract spatial and frequency features from the pre‐processed images. The adaptive fusion module combines the multi‐scale features, which are then fed to the softmax classifier to classify images as benign or malignant. Using the greylag goose optimization (GGO) algorithm, the model's parameters are adjusted to produce better classification results. Next, by merging the cancerous images from The Cancer Genome Atlas (TCGA) dataset and the BreakHis dataset, recurrence is predicted. Recurrence is predicted using clinical and image features. Finally, it evaluated the performance of the proposed framework. The accuracy obtained by the proposed method is 99%, precision is 98.5%, recall is 98%, f ‐score is 98.2%, and MCC is 98%.