Robust nuclei segmentation with encoder‐decoder network from the histopathological images

Abstract Nuclei segmentation is a prerequisite and an essential step in cancer detection and prognosis. Automatic nuclei segmentation from the histopathological images is challenging due to nuclear overlap, disease types, chromatic stain variability, and cytoplasmic morphology differences. Furthermore, it is demanding to develop a single accurate method for segmenting nuclei of different organs because of the diversity in nuclei size, shape, and appearance across the various organs. To address these challenges, we developed a robust Encoder‐Decoder network for nuclei segmentation from the multi‐organ histopathological images. In this approach, we utilize a pre‐trained EfficientNet‐B4 as an Encoder subnetwork and design a new Decoder subnetwork architecture. Additionally, we have applied morphological operation‐based post‐processing to improve the segmentation results. The performance of our approach has been evaluated on three public datasets, namely, Kumar, TNBC, and CPM‐17 datasets, which contain histopathological images of seven organs, one organ, and four organs, respectively. The proposed method achieved an aggregated Jacquard index of 0.636, 0.611, and 0.706 on Kumar, TNBC, and CPM‐17 datasets, respectively. Our proposed approach also shows superiority over the existing methods.