Comparing reliability between different ultrasound techniques

I read with great interest the recently published article by Sur et al.1, in the November issue of the White Journal. In this study the authors intended to compare the intraobserver reliability of embryo volume measurement using a ‘semi-automated technique’—using both Virtual Organ Computer-aided AnaLysis (VOCAL) and Sonography-based Automated Volume Count (SonoAVC)—with a manual technique using VOCAL alone. The authors concluded that the semi-automated technique is more reliable than is the manual technique for embryo volume measurement. However, based on their results, I strongly disagree with their conclusion; additionally I have some minor comments about the study. Although the terms ‘reliability’ and ‘agreement’ are sometimes used interchangeably, they refer to different concepts2. A recent guideline for reporting reliability and agreement studies2, a statistical review3 and even some reliability studies/agreement studies recently published in the White Journal4-6 have reported how reliability and agreement should be defined. Reliability relates the magnitude of the measurement error in observed measurements to the ‘true’ variability between subjects or, in other words, the ability of a measurement to differentiate between subjects or objects2. Agreement quantifies how close two measurements made on the same subject are, regardless of the ‘true’ variability between subjects3. The authors of these studies concur that evaluating both reliability and agreement might be important in some situations, particularly when different samples are being evaluated, which might present different ‘true’ variability (heterogeneity) between subjects. They also agree that the intraclass correlation coefficient (ICC) should be used to evaluate the reliability of methods assessing continuous data (e.g. fetal volume) and that Bland–Altman plots might be used along with other methods when assessing the agreement. In the study of Sur et al.1, the authors described that their objective was to compare the intraobserver reliability between the two methods of assessing fetal volume. Therefore, I expected them to compare the ICCs between these methods, preferably considering the 95% CIs. Since they concluded that the semi-automated technique is more reliable than is the manual technique for embryo volume measurement, I assumed that the ICC for the semi-automated technique was greater than that observed for the manual technique, but this information was not provided in the abstract. When examining the full text I was surprised by the results. Actually, the ICC for the manual technique was higher than that observed for the semi-automated technique: 0.976 (95% CI, 0.926–0.991) vs. 0.942 (95% CI, 0.874–0.997). This result means that, when using the manual mode, 97.6% of the variability in measurements of embryo volumes was estimated to be due to genuine differences between fetuses, 2.4% being due to errors in the measurement process and the observer involved; when using the semi-automated technique, 5.8% of the variability in measurements was estimated to be to due errors in the measurement process and the observer. In other words, random errors were responsible for a greater proportion (2.4 fold) of the total variability when using the semi-automated method. Sur et al. should have concluded that the manual technique is more reliable than the semi-automated one. However, there was an overlap in the 95% CIs between the estimated ICCs, and it is possible that the observed difference occurred only by chance. Yet, the 95% CIs provided were miscalculated: using the estimated ICC and the sample size of 52 subjects, I calculated different values (Table 1). Even considering the new and correct 95% CIs, there is still a small overlap between the two 95% CIs of the estimated ICCs; however, this would probably not occur if a larger sample (> 90 subjects) were examined, providing more precise estimates. Therefore, the best conclusion based on the results of this study is that the manual technique is probably more reliable than the semi-automated one; however, the evaluated sample did not provide sufficient precision for a more robust conclusion. The worse reliability associated with the semi-automated technique is very plausible, since there are more sources of error: each of the three measurements—gestational sac, yolk sac (using the manual method) and amniotic fluid (using SonoAVC)—will provide errors that will be summed when calculating fetal volume. On the other hand, the intraobserver agreement, measured as percentage difference between measurements, seems to be better for the semi-automated method. However, comparing the intraobserver agreement between these two methods is complicated, because the semi-automated technique resulted in larger measurements than did the manual method (three-fold). This huge difference in systematic error probably occurred because SonoAVC underestimated the total fluid. However, based on the results from several studies about validity of volume estimation using VOCAL7-9, it is unlikely that VOCAL had underestimated fetal volume to such a degree. The only expected systematic error when estimating fetal volume with VOCAL would be a 5–10% underestimation of fetal volume if limbs are not included, evaluating only fetal head and trunk volume10, 11. In fact, the observed limits of agreement (LOA) of approximately ± 16% when using the semi-automated technique were worse than the observed LOA of ± 25% when using the manual technique: this represents approximately 0.50 cm3 vs. 0.25 cm3 for semi-automated vs. manual techniques. This is consistent with the findings from ICCs: the semi-automated technique again resulted in larger (two-fold) random errors. The authors would have avoided this misleading result if they had also presented the results from Bland–Altman plots using absolute differences. Datasets. By evaluating a single three-dimensional (3D) dataset twice, the authors did not evaluate the true reliability between methods, because only stable information was analyzed. This might be particularly relevant for the variability of automated estimation of amniotic fluid by SonoAVC. In my opinion, the ‘true’ reliability and agreement of an ultrasound method can only be assessed properly when a whole new examination is performed. In this case (fetal volume), a new 3D dataset should have been acquired, preferably with some minutes between acquisitions to permit fetal movements. Evaluated gestational age (5–9 weeks). I believe that fetal volume measurement is of great value, particularly between 10 and 14 weeks' gestation. At this gestational age, fetal head and trunk volume have been shown to be better than crown–rump length (CRL) to evaluate growth impairment12 and to estimate gestational age10. One reason for this is the fetal attitude: small fetal flexions or deflections may modify CRL measurement, while it is very unlikely to change fetal volume estimation. Until 10 weeks' gestation, fetuses are normally in a neutral position, and the improvement of fetal volume over CRL in determining the ‘true’ gestational age is smaller and unlikely to have any clinical meaning13. Citations from previous studies. The authors stated in their introduction that ‘embryo volume, between 7 and 10 weeks of gestational age, has been shown to be a better predictor of gestational age than CRL in a study of 30 in vitro fertilization (IVF) pregnancies11.’ This information is not accurate: in the cited study10, the authors demonstrated that fetal head and trunk volume was more accurate than was CRL to determine gestational age between 10 and 14 weeks' gestation. In another study13, the authors observed that fetal volume assessed by VOCAL was better than was CRL to estimate gestational age between 7 and 10 weeks. However, in the latter study the authors concluded that the improvement was small and unlikely to have any clinical relevance13, probably for the reasons previously described. Inaccurate information. The authors reported that Blaas et al.11 measured fetal volume using the conventional slicing technique. Actually, Blaas et al. measured the fetal volume using EchoPAC-3D, a completely different technique, which requires delineating segmentation lines all over the surface of the embryo and geometric reconstruction. In my opinion, the EchoPAC-3D method provided the best images (Figure 1) and the feeling that the fetuses were measured accurately. However, this technique is time consuming: the segmentation process took between 20 and 30 min per fetus11, which is somewhat prohibitive in clinical practice. The lack of a final rendered image showing exactly what has been evaluated is one of the main flaws of the described semi-automated method: although a huge systematic error was shown, we were not able to determine from where this extra volume came (e.g. umbilical cord, underestimated fluid estimation). Figure 5. In my opinion, this figure is not very intuitive because it is hard to understand how one measurement could be > 100% smaller than another. The authors should provide an explanation reporting that the difference between measurements is related to the average value between two techniques and not comparisons between the methods. For example, we can see one point near to − 150% on the y-axis and close to 1.0 cm3 in the x-axis. This means that the average value between the two techniques was 1 cm3 and that the difference between the measurements was 1.5 cm3 (150% of 1.0 cm3); therefore, one might conclude that in the same fetus the manual technique measured 0.25 cm3 while the semi-automated technique measured 1.75 cm3 (a seven-fold difference). Including limbs on fetal volume. I agree that including the limbs in the measurement of the fetus will probably result in values closer to the whole fetus, since they represent 5–10% of the total volume between 7 and 12 weeks11 and I understand why the authors developed a strategy for including the limbs before the 10th week when using VOCAL, ‘drawing a thin connecting stalk between the limbs and trunk’. However, this thin stalk does not exist and drawing such stalk might be considered another source of bias, which is acknowledged as a limitation by the authors. In my opinion, including the limbs on fetal volume estimation is not important because the underestimation in volume would be predicted, and be relatively small and constant14. Therefore any observer would be able to compare the measured value with proper reference curves determined using VOCAL or any other method that does not include the limbs on fetal volume estimation. Three-dimensional (3D) surface-rendered fetus delineated with Virtual Organ Computer-aided AnaLysis1 (a) and EchoPAC-3D11 (b). When using the semi-automated technique, combining VOCAL with SonoAVC as suggested by the authors1, the observer is not able to evaluate precisely what has been measured using the rendered images. Therefore, even very large systematic errors (> three-fold) may be overlooked, which should be considered as a significant limitation of this method. Images reprinted with permission. W. P. Martins*, * Departamento de Ginecologia e Obstetrícia da Faculdade de Medicina de Ribeirão Preto, Universidade de São Paulo (FMRP-USP), Av. Bandeirantes, 3900–8° andar - HCRP - Campus Universitário, Ribeirão Preto, São Paulo, Brazil. CEP: 14049-900