Combining fully convolutional networks and graph-based approach for automated segmentation of cervical cell nuclei

Cervical nuclei carry substantial diagnostic information for cervical cancer. Therefore, in automation-assisted reading of cervical cytology, automated and accurate segmentation of nuclei is essential. This paper proposes a novel approach for segmentation of cervical nuclei that combines fully convolutional networks (FCN) and graph-based approach (FCNG). FCN is trained to learn the nucleus high-level features to generate a nucleus label mask and a nucleus probabilistic map. The mask is used to construct a graph by image transforming. The map is formulated into the graph cost function in addition to the properties of the nucleus border and nucleus region. The prior constraints regarding the context of nucleus-cytoplasm position are also utilized to modify the local cost functions. The globally optimal path in the constructed graph is identified by dynamic programming. Validation of our method was performed on cell nuclei from Herlev Pap smear dataset. Our method shows a Zijdenbos similarity index (ZSI) of 0.92 ± 0.09, compared to the best state-of-the-art approach of 0.89 ± 0.15. The nucleus areas measured by our method correlated strongly with the independent standard (r 2 = 0.91).