Analysis and acquisition reproducibility of 3D power Doppler

We read with great interest the article by Lai et al.1, evaluating the intra- and interobserver reproducibility of three-dimensional (3D) power Doppler of the placenta. Based on the results of this work the authors conclude that, ‘There is insufficient evidence to support the meaning, reliability or reproducibility of VOCAL (VI, FI or VFI) as a tool to quantify placental perfusion despite its use in multiple publications and journal submissions’. They go on to say that, ‘there is poor reproducibility at the most fundamental level’ and that, ‘Further investigation into the reproducibility of placental perfusion and quantification using VOCAL is required before development and application as a clinically useful tool’. A recent article by a different group also assessing placental 3D power Doppler reproducibility2, provided contradictory results; the authors concluded that ‘3DPD placental vascular indices obtained using the automatic sphere mode had a good intra- and interobserver reproducibility for pregnancies between 26 and 35 weeks’. One wonders why two recent studies evaluating the same issue reached such different conclusions? Having studied both papers in more detail we wonder if there are some methodological issues that need to be considered, and if the conclusions, especially those of Lai et al.1, may be somewhat premature. Lai et al.1 were absolutely right to consider that data acquisition as the analysis of a single dataset is not truly reflective of clinical practice. The analysis of a single dataset has the potential to produce perfect agreement, as indicated by an ICC of 1.0, as the power Doppler information in that dataset is stable. The serial acquisition of 3D data is much less likely to produce identical datasets, as each will be influenced by natural and unavoidable biological and physiological variation as well as by physical movements of the transducer or the patient. However, one cannot assess and quantify the variation in data acquisition unless we first establish the variation in data analysis and then account for this in the overall analysis. Surprisingly, on consideration of the results of Lai et al.1 for the reliability of data analysis alone, the reported ICC values obtained for both intra- and interobserver measurement reproducibility are relatively low, much lower in fact than any that have been published so far in either obstetrics or gynecology. Either the current literature is falsely reassuring, or the work by Lai et al. is flawed in its design, conduct or interpretation. We hypothesize the problem to lie with Lai et al., as the values reported for intraobserver measurement reliability were far lower than were the values for interobserver evaluation. This makes little statistical sense as the latter should be associated with less agreement. Yet, despite the negative conclusions and poor intraobserver agreement, the interobserver ICCs are indicative of a good level of agreement: all were above 0.753, 4. The results published by Guimaraes Filho et al.2 were more compatible with the current literature, with ICCs of 0.92, 0.99 and 0.90 for intraobserver comparisons and ICCs of 0.75, 0.98 and 0.86 for interobserver comparisons for VI, FI and VFI, respectively. However, only reproducibility of data analysis was evaluated in this latter study. Another important point, and one that may explain the wide variation seen in the two studies, relates to the maintenance of Doppler settings, which is essential if one is to compare datasets5. In the work of Lai et al.,1 Doppler gain was subjectively adjusted as follows: ‘the 3D power Doppler signal was optimized by increasing the Doppler gain until the point of color blooming, then reducing to show only flow signal’. Color gain is directly correlated with the 3D ‘vascular indices’ derived using the histogram and VOCAL6 and the use of a gain above 0 has been shown to cause artifactual signals even in the total absence of flow5. The difference between the two observers may have been caused by different color gain adjustments between the observers, therefore, which would have been avoided by selection of a fixed color gain setting (e.g. 0 dB for the Voluson 730 Expert ultrasound machine used in this study). Moreover, when considering interanalyzer variability, it is unclear how the two analyzers evaluated the datasets: did they both evaluate a single dataset per quadrant of placenta, or did Analyzer 1 evaluate the dataset from Acquirer 1 and Analyzer 2 evaluate the dataset from Acquirer 2? If they did not evaluate the same dataset, then reproducibility of both analysis and acquisition were examined and not just reproducibility of analysis as implied. Additionally, we are confused as to why the authors used different statistical tests to evaluate reproducibility of data analysis, data acquisition and regional variability. If they had restricted their assessment to one test, such as intraclass correlation, the reader would have been able to compare the effect of analysis, acquisition and regional variations in the variability of 3D power Doppler indices. If a more detailed analysis was required, and in our opinion it was, Bland and Altman plots and limits of agreement7 should have been estimated3 to demonstrate the expected variation in the absolute or relative values for each index. The combined use of limits of agreement and intraclass correlation is well described and recommended for the evaluation of reproducibility of continuous scale data3, but we are unclear as to what additional information the paired t-test and coefficient of variation provided. It would also have been helpful to provide information in Table 1 to show how these values are specifically related to VI, FI and VFI. Lai et al. may be right that 3D power Doppler is not a reliable, or indeed appropriate, test with which to analyze blood flow or the vascularity of the placenta—at least, not in their hands due to their own very low measurement reliability. By their own admission, the analysis of a single dataset has the potential to generate comparable results both within and between observers as the information is stable and they should ask themselves why they were unsuccessful when so many other authors have managed to do so. Unfortunately, whilst we totally agree that consideration and assessment of the process of data acquisition is essential, Lai et al., were unable to examine this in their study due to their unacceptably low reliability of data measurement. We feel their conclusions relate more to their own practice and should be interpreted with caution before other researchers abandon the technique. Future studies evaluating the reproducibility of 3D Doppler of the placenta are still necessary; these studies should use standardized settings and consider the effect of both acquisition and analysis. W. P. Martins* , N. J. Raine-Fenning , * Departamento de Ginecologia e Obstetrícia da Faculdade de Medicina de Ribeirão Preto, Universidade de São Paulo, Av. Bandeirantes, 3900 - 8° andar - HCRP - Campus Universitário, Ribeirão Preto, São Paulo, Brazil, Escola de Ultra-sonografia de Ribeirão Preto, Ribeirão Preto, Brazil, Nottingham University Research and Treatment Unit in Reproduction (NURTURE), Division of Human Development, School of Clinical Sciences, University of Nottingham, Nottingham, UK