A generalizable deep learning system for cardiac MRI

Figures

• 

  Fig. 1 |. Project overview. a, Large-scale contrastive pre-training. CMR cine sequences in the form of video data are fed to a video transformer network, and the paired text reports are fed into a parallel text transformer network. The embeddings produced by each (V, video; T, text) represent compressed representations of the complex inputs to each network. The networks are trained to maximize agreement between true pairs of video and text that arise from the same CMR scan (that is, the diagonal of the similarity matrix highlighted: V1T1, V2T2 … VnTn) and minimize the agreement between samples that come from different scans. b, In the process in a, we see that as training progresses, more complex local and global structural features emerge in the video embeddings (V). The scans initially viewed as an indistinct amalgam in high-dimensional space at the start of training (epoch #1) begin to develop separations and localized clusters that are more distinct by epoch #100. c, Evaluating the ability of the network, on external datasets, in separating diseases or probing the UK BioBank as described in later sections is achieved by freezing all the parameters of the video encoder (at this point,…

• 

  Fig. 2 |. Contrastive pre-training enables learning of demographic and pathophysiological representations. 2D t-SNE plots of low-dimensional embeddings generated from a forward pass on 34,490 unique scans from the UK BioBank (n = 31,693 unique individuals) using a Kinetics-400 pre-trained checkpoint (baseline, right column) compared to a contrastive pre-trained checkpoint (ours, left column). The colour labels help with understanding the basis of clustering. While at first glance it may appear that the Kinetics baseline model produces a set of 5 readily separable clusters vs the more homogeneous-appearing plot for the contrastive pre-trained model, the basis of the baseline Kinetics-initialized clusters are simply the various MRI view planes. On further exploration we find that the Kinetics-400-generated embeddings fail to capture the required information to separate low-ejection fraction states, cardiomyopathy, gender or age (lack of separation of colour for each variable). This is different from the contrastive pre-trained generated embeddings that show clear demarcations by physiologically relevant characteristics, allowing for zero-shot separation of low-ejection fraction states,…

• 

  Fig. 3 |. Clinical-grade LVEF% regression using contrastive pre-trained vision encoder. a, Dashed horizontal line represents Kinetics-400 baseline performance for the problem of LVEF% regression with 100% of the UK BioBank training data (for fine-tuning and transfer-learning modes). Each point represents validation-set results from a randomly initialized subset (n = 5 independent replicates) of either 1% or 10% of the available training data (x axis). Superior performance (lower is better) is seen with fine tuning, whereas unfreezing all mViT layers (transfer learning) is detrimental to performance. Fine tuning with just 1% of available data yields superior validation results compared to the Kinetics-400 baseline. Box limits represent the interquartile range (IQR), whiskers 1.5× IQR, and the horizontal marker is the median. b, Bland–Altman (BA) plots for test-set results on the UK BioBank and an additional external test set using the Kaggle Dataset. MAE on UK BioBank test set is 3.344 (s.d. 3.615), with BA limits of agreement of −9.91% to +9.61%. Kaggle test results: MAE of 6.880 (s.d. 5.309), with BA limits of agreement of −18.7% to +8.03%. c, Test-set performance of Kinetics-400 baseline…

• 

  Fig. 4 |. Data-efficient disease diagnosis from cine sequences alone. a, Test-set AUCs on the internal dataset for contrastive pre-trained (magenta) and Kinetics-400 baseline (light pink) checkpoints for models fine tuned for the binary classification problem of disease diagnosis. Disease labels are placed along the x axis, with test AUC on the y axis. The dashed grey line marks an AUC of 0.5, the performance expected from a random classifier. Solid grey connecting lines mark the performance difference between contrastive pre-trained and baseline models. Remarkable classification performance is observed for conditions such as hypertrophic cardiomyopathy, amyloidosis, systolic anterior motion of mitral valve and dilated cardiomyopathy, whereas models somewhat struggle for conditions that typically require gadolinium contrast or additional scanning sequences for diagnosis, such as differentiating cardiac masses from cardiac thrombus. The numbers of patients with disease for each disease label are listed along the x axis (n = positive classes), in black text within grey-outlined box. b, Test-set results of the contrastive pre-trained (navy blue) and Kinetics-400 baseline (light blue) models…