An update on the analysis of agreement for orthodontic indices

The training of clinicians in the correct use of commonly used orthodontic indices involves calibration. The level of agreement between the trainee and a standard is assessed both as a measure of reproducibility and the success of training programmes. For the Peer Assessment Rating (PAR) index and the Index of Complexity, Outcome and Need (ICON), the recommended level of acceptable inter-rater agreement is no more than +/-12 and +/-18, respectively. Many commonly used methods of analysing this type of agreement are inappropriate. The method used in this investigation allows the calculation of limits of agreement, which easily demonstrate any major departures in agreement between trainee scores and standard scores. The basic method assumes that the differences between trainee and standard scores are normally distributed and that there is no relationship between these differences and the magnitude of the index. An extension to this approach is required when the assumptions of the basic method are not upheld. This extension provides a regression-based approach to calculating limits of agreement. The results of this study demonstrate that the assumptions of the basic approach need to be checked for each comparison of trainee versus standard. In addition, regression-based methods are a more accurate means of calculating limits of agreement when these assumptions are not upheld. They also provide more information about bias and the range of disagreement between raters.