We consider the problem of constructing a confidence interval for the intraclass correlation coefficient in an interrater reliability study when the raters are assumed to be randomly selected from a population of raters. A Monte Carlo simulation study is conducted to investigate the true coverage probabilities of the commonly used intervals proposed by Fleiss and Shrout, which rely on Satterthwaite's two-moment approximation. These intervals are shown to be substantially anticonservative in certain cases. We propose intervals based on higher-moment approximations obtained using the Pearson system of distributions. The modified intervals are more conservative and generally more satisfactory than those obtained by the two-moment approximation. The competing methods are illustrated using data from a study of the natural history of facioscapulohumeral muscular dystrophy.
No takes yet. Share an insight, caveat, or question.
Zou et al. (1999) studied this question.