Measures of Interobserver Agreement

This is quite a short and a tight book, with 102 pages excluding worked examples (workshops), references and the index. It covers both continuous and categorical data, two or more raters and also more than one rating per individual rater (the ratings are then dependent), finishing by discussing the issue of sample size requirements for the design of a rater reliability study. Likelihood and generalized equations models for dependent data are also covered. It covers mainly clinical settings and is strongly example based with many exercises using SAS code worked through. These can also be downloaded from the Web site (http://www.crcpress.com—follow ‘electronic products’ links), though the data are apparently not there. For continuous measures (Chapter 2), the natural measure of agreement is shown to be the intraclass correlation, interpretable in turn as the proportion of variance that is attributable to the object of measurement. Three cases are distin-guished according to whether each subject is rated by the same or by different raters, and whether or not the raters are themselves sampled randomly from a large population. Expressions and inference procedures are given for each situation. An important warning is given about the disadvantages of using the product moment correlation coefficient between two methods of measurement as a measure of agreement, and the value of Bland and Altman's graphical method for diagnosing problems resulting from unequal variances of these measures. The main part of the book (Chapters 3–5) deals with the case when the assessments are either categorical or ordinal. After comparing alternative indices of adjusted agreement, the κ chance-corrected measure is studied in detail, including the stratified case, and warnings are given about its interpretation. This is then generalized to the case of more than two raters, and to more than one rating per observer. The final chapter addresses the determination of sample size for the design of a reliability study for both continuous (normal and non-normal) and dichotomous measurements, and includes a consideration of cost through minimizing a function of variance and costs. Interestingly, the overall precision of the estimated reliability index tends to be maximized in most cases by using a small number of replicate measurements per subject (and so a larger number of subjects). Be warned that, although the proof-reading has generally been careful, all the references in the index to the bibliography are around two pages out. Mohamed Shoukri is well placed to write this book, as he is one of the foremost current researchers in this field. I recommend it.