A multi‐model approach to simultaneous segmentation and classification of heterogeneous populations of cell nuclei in 3D confocal microscope images

Automated segmentation and morphometry of fluorescently labeled cell nuclei in batches of 3D confocal stacks is essential for quantitative studies. Model-based segmentation algorithms are attractive due to their robustness. Previous methods incorporated a single nuclear model. This is a limitation for tissues containing multiple cell types with different nuclear features. Improved segmentation for such tissues requires algorithms that permit multiple models to be used simultaneously. This requires a tight integration of classification and segmentation algorithms. Two or more nuclear models are constructed semiautomatically from user-provided training examples. Starting with an initial over-segmentation produced by a gradient-weighted watershed algorithm, a hierarchical fragment merging tree rooted at each object is built. Linear discriminant analysis is used to classify each candidate using multiple object models. On the basis of the selected class, a Bayesian score is computed. Fragment merging decisions are made by comparing the score with that of other candidates, and the scores of constituent fragments of each candidate. The overall segmentation accuracy was 93.7% and classification accuracy was 93.5%, respectively, on a diverse collection of images drawn from five different regions of the rat brain. The multi-model method was found to achieve high accuracy on nuclear segmentation and classification by correctly resolving ambiguities in clustered regions containing heterogeneous cell populations.