Assessing Agreement between Methods of Measurement

Featured Article: Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986;i:307–10.3 We began our careers as medical statisticians in the 1970s, working in an epidemiology department. The quality of measurements was well-recognized in epidemiology, as illustrated by the development of the “random-zero” sphygmomanometer to enable unbiased measurement of blood pressure. But it was after we had moved on to separate clinical research environments that we independently faced many scenarios relating to how to measure biological quantities, initially in the fields of rheumatology and cardiology. We discovered that the standard way in which measurement studies were then analyzed was by correlation coefficients, whether authors were comparing measurements obtained by different observers using the same method or measurements obtained using different methods. It was obvious to us that correlation did not provide a useful answer to the agreement between 2 methods of measurement. For example, we could double the size of all the observations made using one of the methods without altering the correlation coefficient. The question was not whether the 2 methods agreed but rather how closely they agreed, as the differences between the pairs of measurements would be much more informative. Accordingly, we developed some simple ideas based around the mean and SD of the differences, from which we could estimate a range within which we would expect a high percentage (usually 95%) of observed differences to lie—we called these “limits of agreement.” These limits depend on the assumptions that the mean and SD of the differences are unrelated to the magnitude of the measurement and that they follow an approximately normal distribution. We thought that a plot of the data would be helpful to check these assumptions. Specifically, we suggested a histogram of the differences between the paired measurements and a plot of the differences vs their mean. We saw this plot, now often called a “Bland–Altman plot,” as giving support for the numerical summary, not as the method itself (1). We first presented our thoughts at a conference of statisticians in Cambridge in 1981, and our paper based on the talk appeared in 1983 (2). This paper was inaccessible to our clinical colleagues, so we thought it would be useful also to publish the ideas in a medical journal. Thus, in January 1986, our article was published in The Lancet. It is astonishing that it has become one of the most cited statistical papers ever published. Our article in The Lancet included a data set we collected particularly for the study, consisting of replicate measurements of peak expiratory flow using 2 different instruments—the sample included colleagues, family, and ourselves (we were the 2 people with the best lung function). It was obvious to us that it would be beneficial to include a table of the raw data—however, this practice remains rare despite the current focus on transparency. We certainly were not the first people to recognize that correlation coefficients did not assess agreement but rather association. Notably, we quoted Westgard and Hunt, who wrote wisely in Clinical Chemistry in 1973, “The correlation coefficient … is of no practical use in the statistical analysis of comparison data” (3). Their preferred approach used prediction intervals based on least squares regression. That the topic is of critical importance in clinical chemistry is illustrated by the fact that this was 1 of 5 references in our 1983 paper to articles in chemistry journals. At the time we thought our simpler approach was obvious and so our ideas were unlikely to be original, but we could not find a similar proposal. We since found that Donald Mainland had made similar comments about correlation 40 years earlier (4), but we still have not found the same suggestion about how to compare methods of measurement. Since the first paper, we have extended the approach in various ways, in particular to allow analysis of replicated measurements (5, 6). Method comparison studies are clearly a topic of permanent relevance. Our 1986 paper is the most cited article in The Lancet and is still being cited several times a day, but our other related papers are the most cited articles in several other journals (2, 5, 6). Measurement is fundamental to clinical practice and research. After >40 years as medical statisticians, we remain disappointed that too few people realize that.