Assessing Deep Hybrid Representations for Colorectal Polyp Classification in Histopathology Images

This study investigates the use of Convolutional Neural Networks (CNNs) and Vision Transformers (ViT) backbones for feature extraction from histopathological images, and further evaluates the effectiveness of combining features from different depths of both architectures. The backbones were employed to extract features from colorectal polyp images in the MHIST dataset. Features obtained from each model individually, across different deep layers, were also combined and subjected to a dimensionality reduction process. Subsequently, both the individual and combined feature sets, with and without dimensionality reduction, were evaluated using linear classification models. The combined features achieved an accuracy of 85.55%, surpassing all results obtained from individual models and other approaches. These findings show that features from the first and middle layers, and feature fusion enhances robustness by leveraging the complementary strengths of each architecture.