A novel shape prior based segmentation of touching or overlapping ellipse-like nuclei

Cell nuclei segmentation is a key issue in automatic cell image analysis for nuclear malignancy. However, due to the complexity of microscopic images, it is usually not easy to obtain satisfied segmentation results, especially on the separation of touching or overlapping nuclei. We propose a method to separate overlapping nuclei whose shapes are similar to ellipses, even if they are tightly clustered and no edge is present where they touch. As a class-specific approach, it introduces a statistical shape model as an extra constraint within the energy functional that measures the homogeneity of regional intensity. The desired contours of each nucleus can be obtained by minimizing this energy functional. The proposed algorithm has been tested on human cervical nuclei images. Experiment results show that our method can separate touching or overlapping ellipse-like nuclei from each other accurately, and the tests on noisy and textured nuclei images also demonstrate its robustness. The resulting segmentation contours are ellipses in different sizes and directions, therefore the shapes of the nuclei have been preserved to a certain degree. The algorithm can be naturally extended to color images, and also has the potential to deal with the separation for overlapping nuclei of other shapes.