Interpreting reproducibility results for ultrasound measurements

We are frequently concerned about problems in the interpretation of reproducibility studies for ultrasound measurements, as when reading an article recently published in the Journal1. Besides the fact that reproducibility study reports are often incomplete2, there are other important problems that need to be addressed. Intraclass correlation coefficient (ICC) and concordance correlation coefficient (CCC) are popular methods used to assess reliability2. These measurements of reliability intend to estimate the proportion of the total variance that might be attributed to the ‘true’ variance between subjects; for example, a CCC value of 0.90 would mean that 90% of the total variance was due to differences between subjects while the other 10% was probably caused by observer error. Of these two methods, we prefer CCC, because there are several models for ICC – choosing the most appropriate method could be complicated – and because ICC is only valid when ANOVA assumptions are present, which is frequently not the case3. One important problem regarding the interpretation of reliability is that authors often interpret its results using cut-off values from guides designed for other purposes/indices (e.g. correlation coefficient, kappa coefficient), which are lower than are values appropriate for examining reliability results2, 4. Any interpretation based on such values is therefore misleading and creates false confidence in the method, which might jeopardize both future research and clinical practice. One should consider using even higher cut-off values when interpreting the results for fetal measurements, as the ‘true’ variance between fetuses across a wide range of gestational ages is much larger than is that observed when measuring adults. However, when only a narrow gestational age is considered (e.g. from 11 + 0 to 13 + 6 weeks), the more widely used lower cut-offs may be appropriate. Our suggested cut-off values are presented in Table 1. The method known as limits of agreement (LoA) is frequently used, and should be interpreted as the interval expected to contain 95% of the differences between measurements. Although this seems to be quite simple, its interpretation is very dependent on the clinical relevance of the findings, particularly when examining the absolute difference between measurements. We therefore suggest using the relative difference between measurements (i.e. a percentage) instead of absolute difference, as its interpretation is easier to generalize (Table 1). Another important flaw in many reproducibility studies is that they only examine the difference observed when repeating measurements using the same stored images or datasets instead of performing a completely new examination1. In so doing, there are important sources of variability that are not taken into account; for example, fetal/bowel movements and differing pressure applied on the probe5. Therefore, using such a strategy to examine reproducibility is misleading, as the results obtained will be better than those expected in clinical practice. The precision of estimates (95% CIs) should always be considered when interpreting results. Although our suggested cut-off values might be considered arbitrary and may not fit all situations, we believe they will help inexperienced authors to improve their interpretation of reproducibility results. W. P. Martins* and C. O. Nastri Department of Obstetrics and Gynecology, Medical School of Ribeirao Preto, University of Sao Paulo (FMRP-USP), Ribeirao Preto, Brazil; Faculty of Technology in Health - Ultrasonography School of Ribeirao Preto (FATESA-EURP), Ribeirao Preto, Brazil *Correspondence. Av. Bandeirantes, 3900, 8 andar, HCRP, Campus Universitário, Ribeirao Preto, Sao Paulo 14048–900, Brazil (e-mail: [email protected])