Predicting post-operative right ventricular failure using video-based deep learning

Figures

• 

  Fig. 1. Overview of the project. a Pre-operative echocardiography videos are processed as a stack of 32 frames. A two-stream implementation of raw greyscale videos and optical flow channels are fed into a 3D convolutional neural network to produce the prediction of RV failure. b The clinical workflow for predicting future risk of RV failure begins in the pre-operative phase using a combination of clinical parameters and a detailed echocardiographic assessment. Risk scores such as the CRITT and Penn scores are calculated thereafter to aid in risk stratification following which a decision is made to either electively implant a concomitant RVAD or proceed with LVAD alone. c The clinical ground truth is determined largely by the persistent need for inotropes past post-operative day 14 or right ventricular mechanical circulatory assist devices during the post-operative recovery period. MCS-ARC definitions are detailed in Supplementary Table 1. Artwork attribution from left to right in (b): Wikimedia Commons by Videoplasty.com; Anton Kalashny; Ralf Schmitzer".

• 

  Fig. 2. Performance of the AI system, clinical risk scores, and clinical benchmarking. a ROC curve of the AI system compared to contemporary clinical risk scores. The performance of the AI system was 0.729 (95% CI 0.623–0.835). b ROC curves of clinical expert team and independently calculated metrics of right ventricular function compared to the AI system. The performance of the AI system was found to exceed both clinical experts and the traditional risk scoring systems. LV-ESA left ventricular end systolic area, RVED-Area right ventricular end diastolic area, RVEF RV Ejection Fraction, RVES-Area RV end systolic area.

• 

  Fig. 3. Analysis of saliency maps from pre-operative echocardiograms. Representative input videos and visualizations for both systolic and diastolic phases of the cardiac cycle across patients with and without RV failure, in the form of a confusion matrix. True positives (bottom right quadrant), false positives (bottom left quadrant), true negatives (top left quadrant), and false negative examples (top right quadrant). Colour scale for each quadrant represents regions that contributed most to the predicted class (red) and those that pushed predicted probability away from the predicted class (blue).