Hybrid Deep Learning Architectures for Histological Image Segmentation

In histopathology image analysis, accurate segmentation of nuclei holds immense significance, particularly in the early detection and treatment of diseases like breast cancer. Nuclei segmentation is a fundamental but challenging task because of the intricate variations in nuclear shapes, sizes, densities, and overlapping instances. In this paper, we evaluate eight convolutional neural network (CNN) models, two of them existing models namely U-Net, SegNet, and six hybrid models by combining U-Net and SegNet modify decoder with ResNet, VGG and DenseNet (ResNet-UNet, ResNet-SegNet, VGG-UNet, VGG-SegNet, DenseNet-UNet, and DenseNet-SegNet. This experiment aims to identify the best deep-learning model for segmenting hematoxylin and eosin (H&E) stain images using a publicly available dataset called MoNuSeg. From the experimented work, we found that VGG-UNet outperforms other models with an F1 score of 0.8452 and IoU of 0.6929 respectively. This research will serve as a foundation for the future construction of more complex deep learning models with cascade or any combination of the models studied.