A Hybrid Deep Learning and Machine Learning Approach with Mobile-EfficientNet and Grey Wolf Optimizer for Lung and Colon Cancer Histopathology Classification

BACKGROUND: Lung and colon cancers are among the most prevalent and lethal malignancies worldwide, underscoring the urgent need for advanced diagnostic methodologies. This study aims to develop a hybrid deep learning and machine learning framework for the classification of Colon Adenocarcinoma, Colon Benign Tissue, Lung Adenocarcinoma, Lung Benign Tissue, and Lung Squamous Cell Carcinoma from histopathological images. METHODS: Current approaches primarily rely on the LC25000 dataset, which, due to image augmentation, lacks the generalizability required for real-time clinical applications. To address this, Contrast Limited Adaptive Histogram Equalization (CLAHE) was applied to enhance image quality, and 1000 new images from the National Cancer Institute GDC Data Portal were introduced into the Colon Adenocarcinoma, Lung Adenocarcinoma, and Lung Squamous Cell Carcinoma classes, replacing augmented images to increase dataset diversity. A hybrid feature extraction model combining MobileNetV2 and EfficientNetB3 was optimized using the Grey Wolf Optimizer (GWO), resulting in the Lung and Colon histopathological classification technique (MEGWO-LCCHC). Cross-validation and hyperparameter tuning with Optuna were performed on various machine learning models, including XGBoost, LightGBM, and CatBoost. RESULTS: The MEGWO-LCCHC technique achieved high classification accuracy, with the lightweight DNN model reaching 94.8%, LightGBM at 93.9%, XGBoost at 93.5%, and CatBoost at 93.3% on the test set. CONCLUSIONS: The findings suggest that our approach enhances classification performance and offers improved generalizability for real-world clinical applications. The proposed MEGWO-LCCHC framework shows promise as a robust tool in cancer diagnostics, advancing the application of AI in oncology.