Verity plots: A novel method of visualizing reliability assessments of artificial intelligence methods in quantitative cardiovascular magnetic resonance

Figures

• 

  Fig 1. Classical Statistical Plots. Illustrative examples for correlation, box and Bland-Altman plots with capabilities and limitations listed on the right. Legend: PCC: Pearson’s correlation coefficient, stddev: standard deviation.

• 

  Fig 2. Statistical Definitions and Relationships. On the left different comparisons are shown. Intra- and interreader comparisons (same and different readers) are used to calculate tolerance ranges (BTR and VTR). The BTR/VTR contains 95% of evaluation differences for an intrareader/interreader comparison. The AI’s performance is acceptable if its confidence interval (CI) is covered by the BTR, and its tolerance interval (TI) is covered by the VTR. The CI covers the bias with a confidence of 95%, the TI covers a configurable portion (i.e., 1−α) of the differences with a confidence of 95%. These relationships are depicted in the Reliability Plot, in which the CI (red diamond) leaves the BTR (grey area), but the TI (blue lines) is covered by the VTR (dark grey lines). Legend: BTR: bias tolerance range, VTR: variance tolerance range, CI: 95%-confidence interval, TI: tolerance interval, std: standard deviation, z1+p2: critical value of normal distribution of the cumulative probability (1 + p)/2, χ1−α,υ2: critical value of the chi-square distribution with \/9υ degrees of freedom that is exceeded with probability α.

• 

  Fig 3. Extensive and Condensed Verity Plots. The extensive Verity plot (left) visualizes a measurement comparison. The left half shows a reliability and a density plot. The reliability plot shows the results of the statistical test as well as its central components: the bias’ confidence interval, the bias tolerance range (BTR) around zero, the variance tolerance interval (TI), and the variance tolerance range (VTR) as two grey horizontal lines. Blue depicts acceptability, red unacceptability. The right half shows a Bland-Altman style difference plot, in which the bias and the tolerance interval are shown. The condensed Verity plot (right) for multiple comparisons (Method 1–4) shows a density, strip, and reliability triplet per method comparison.

• 

  Fig 4. Utility of Density Function for Outlier Detection in Verity Plots. The top two figures depict three AI-expert comparisons in condensed Verity plots (left), and of the first AI to the expert in the extended Verity plot (right). Density and point plots reveal systematic outliers (a, b). In the right plot the linear trend of the outliers indicate a systematic failure (b). The systematic outliers break the normal distribution and invalidate the test. These segmentation failures were removed to produce the two plots below. The data are normally distributed and the first two AI methods have acceptable biases and variances. Legend: BTR: bias tolerance range, VTR: variance tolerance range, TI: tolerance interval.

• 

  Fig 5. Plot Comparisons: Verity, Correlation, Box and Bland-Altman Plots. Three quantitative parameters: T1 Midventricular Average, RV EDV, and RV EF (rows 1–3) illustrate differences in expressiveness between Verity, correlation, box and Bland-Altman plots (columns 1–4). Verity plots depict the acceptability of bias and variance clearly, with negative trends visibly accessible in RV EDV and RV EF. Correlation plots show correlations and a zero difference line. PCCs are visible, biases and trends are difficult to see. Box plots offer the median, the spread (50% of differences) as the box’s extent, and 98% of differences as whiskers. The data spread is visible, but trends and outliers are concealed. Bland-Altman plots show the bias, and 1.96*standard deviations as agreement limits. They show the bias and variance of data, as well as trends and outliers. Legend: BTR: bias tolerance range, VTR: variance tolerance range, TI: tolerance interval, PCC: Pearson’s correlation coefficient, Midv.: midventricular, RV: right ventricle, EF: ejection fraction, EDV: end-diastolic volume.

• 

  Fig 6. Customization Options of Verity Plots. Condensed Verity plots can be plotted horizontally (A) to focus attention on the bias and variability around the vertical zero difference line. In (B) the density plots were removed to focus on the statistical test; the spread of differences is presented with the plotted points. In (C) the Verity plot’s density plot was merged with the reliability plot, colours were changed to red. In (D) the points were removed, and the density plot was merged with the reliability test. In (E) the density plots were replaced with box plots, and the reliability tests were removed. In (F) the density plot was moved entirely to the left, and colour coding was chosen separately. Legend: BTR: bias tolerance range, VTR: variance tolerance range, TI: tolerance interval, AI: artificial intelligence.