Harnessing feature extraction capacities from a pre-trained convolutional neural network (VGG-16) for the unsupervised distinction of aortic outflow velocity profiles in patients with severe aortic stenosis

Figure 1. General information about the study population from recruitment to follow-up. (A) A flowchart for patient recruitment in order to select 101 patients with best quality Doppler tracings. Notably, 56 out of 366 patients (15.3%) had no Doppler tracings as raw data available. (B) Kaplan–Meier survival plot testing for differences in survival between derivation and validation cohorts. RHC, right heart catheterization; TAVR, transcatheter aortic valve replacement.

Figure 2. Pre-processing of aortic outflow velocity profiles from patients with severe AS data input for the convolutional neural network. (A) Schematic of the image pre-processing pipeline. One representative aortic outflow velocity profile per patient was extracted from records. Cropping of the region of interest, i.e. during systole, was done manually. Since original echocardiographic images were recorded at different scales, but homogeneity of data input had to be provided, Doppler tracings were further manually scaled according to uniform time and velocity axes (see also Supplementary material online, Figure S1 for a standard operation procedure explaining additional details to create the desired normalized profiles). Neither image normalization nor histogram equalization was applied during pre-processing. Re-sizing to 224 × 224 pixel format as the default input size of the VGG-16 model was already part of the image processing R code after loading the folder with 101 scaled Doppler tracings. (B) Representative Doppler tracings that were excluded due to insufficient contrast, or due to labelling within the aortic outflow velocity profile.

Figure 3. A convolutional neural network followed by PCA and unsupervised k-means clustering provides the proof-of-principle that two subgroups of patients with severe AS can be distinguished according to the aortic outflow velocity profile. (A) VGG-16 network architecture (schematic). The VGG-16 network can be split into two parts: 13 convolutional layers constitute the first part, through which each image is passed through for feature extraction. The convolutional layers are followed by three fully connected layers for classification, and the last layer uses a softmax activation function for final class prediction. Since the aortic outflow velocity profiles were no established class within the ImageNet data set, the classification part of VGG-16 was omitted after pre-training, and hence only the model’s feature extraction capacity was exploited in order to transform aortic outflow velocity profiles to 1D arrays (flatten layer), which were subsequently used for unsupervised clustering. (B) PCA of 1D arrays from 101 aortic outflow velocity profiles. (C) Scatter plot including 95% confidence ellipse in order to illustrate cardiac output and mean aortic valve gradient in accordance with…

Figure 3. (Continued)

Figure 4. Conventional dichotomization of the study population in accordance with elevation in mean aortic valve gradient. (A) Scatter plot illustrating cardiac output and mean aortic valve gradient after dichotomization in accordance with elevation in mean aortic valve gradient. (B) Kaplan–Meier survival analysis in accordance with elevation in mean aortic valve gradient. AVGmean, mean aortic valve gradient.

Figure 5. A flowchart illustrating the application of SMOTE to create a balanced data set for training of the extreme gradient boosting algorithm. CNN, convolutional neural network; SMOTE, synthetic minority over-sampling technique; XGB algorithm, extreme gradient boosting algorithm.

Figure 6. An extreme gradient boosting algorithm opens the perspective to assign patients to beforehand defined clusters by a comprehensive set of functional and structural parameters of cardiac and pulmonary circulatory conditions. (A) Bee swarm plots for comparison of key characteristics between clusters after over-sampling (SMOTE). (B) Confusion matrix (test set). (C) Shedding light on the black box of extreme gradient boosting algorithm-mediated cluster assignment by calculating SHAP (SHapley Additive exPlanations) values for its input variables. The y-axis represents the input variables in descending order of global feature importance, whilst the x-axis indicates the adjustment to the predicted cluster. Moreover, each dot in this sina plot represents an observation, i.e. a patient from the derivation cohort, and the gradient colour denotes the value of the respective input variable. Therefore, if the dots on one side of the central line are increasingly yellow or purple, that suggests that increasing values or decreasing values, respectively, move the predicted cluster in the respective direction (left: Cluster 1; right: Cluster 2). For instance, higher values of AVGmean (purple dots)…