Generating Synthetic Labeled Data From Existing Anatomical Models: An Example With Echocardiography Segmentation

Fig. 1.. Using anatomical models as high quality ground truth annotations, we propose a pipeline to generate large synthetic datasets for training convolutional neural networks.

Fig. 2.. Overview of proposed pipeline implemented for echocardiography segmentation. a) Extraction: pseudo images and ground truth labels are built from the 3D anatomical models. First, a larger cohort of shapes is generated by building a statistical shape model from the original anatomies and sampling new 3D instances using principle component analysis (PCA). Next, 2D slices of the desired view (apical four chamber shown) are sampled. Finally, pseudo-ultrasound images and the corresponding labels are built. Each step expands the size of the dataset. b) Transformation: The pseudo images and a dataset of unlabeled real echo images are used to train a CycleGAN to transform the pseudo images into synthetic ultrasound images. c) Learn: The paired synthetic ultrasound images and model labels are used to train a U-Net segmentation network. d) Test: The network trained on synthetic images is tested on real images to evaluate the utility of the pipeline. The creation of new shape models as well as the transformation module are optional extensions. The slicing can be performed on the original models and the segmentation network can be trained using pseudo images instead. We evaluate the…

Fig. 3.. Example application: echo segmentation. a) Apical two chamber (top) and apical four chamber (bottom) views as shown in an anatomical model. The right images show example real apical two chamber (b) and apical four chamber (c, d) echo images with task labels. (b, c) show the full heart while d) is zoomed to focus on the left ventricle.

Fig. 4.. Extraction details for echocardiography. To extract pseudo images a) a 2D plane is defined from a set of landmarks, b) the plane is rotated and cropped to match standard acquisition parameters and positioned to match standard positioning in real images, c) random noise and shadows are added and the slice is blurred yielding d) the final pseudo image. Additional details are given in Appendix A.

Fig. 5.. Synthetic images closely match real images. Can you guess which images are synthetic? Answers below.3

Fig. 6.. Expert annotations on synthetic images match the anatomical model annotations at a level equal to inter-observer error on real images: A sample image with included labels from the Camus and SiteA images used for the inter-observer study, chosen by taking the median Dice score between \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} }{}${O}_{{2}}$ \end{document} and the original labeler. For the real datasets the original labeler was [30] and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} }{}${O}_{{1}}$ \end{document} for Camus and SiteA respectively. For the synthetic datasets the original label comes from the anatomical models.

Fig. 7.. Networks trained on synthetic data produce accurate segmentations in most cases: Worst, median, and best LVendo segmentation results on the Camus, EchoNet, SiteB, and Pathological test sets for the network trained on the baseline real data and the synthetic data. The task for all networks was LVendo segmentation in apical four chamber end diastole images. Images were ranked by Dice score for the network trained on synthetic data. The baseline and synthetic networks are always specific to the dataset (so for Camus the baseline network was trained on Camus and the synthetic network was trained on synthetic Camus). The yellow arrow points to a bulging septum in that image (see text). The baseline for SiteB and Pathological was SiteA.

Fig. 8.. Shape variations are mainly seen in the LA, not the LV: Four chamber slices showing ± 2 standard deviations from the mean model for the first three modes calculated using principal component analysis (see Sec. II-A.1). LV = left ventricle, LA = left atrium, RV = right ventricle, RA = right atrium, and Myo = myocardium.

Fig. 9.. Annotation bias can yield large differences: Violin plot showing the amount of LVepi area comprised of the LVendo area for the labels in each dataset. A lower value indicates a thicker myocardium. EchoNet does not have LVepi labels.