Comparing the Joint Agreement of Several Raters with Another Rater

In laboratory proficiency surveys which use reference laboratories for the evaluation of participant laboratories, a measure of the agreement of the participant laboratory with the reference laboratories is needed which considers the extent of agreement (or disagreement) among the reference laboratories themselves. In, a measure of nominal scale agreement, is proposed. In is interpreted as follows: Let a specimen be selected at random and rated by a reference laboratory which itself has been randomly selected from the n reference laboratories. If the specimen was also rated by the participant laboratory, this second rating would agree with the first at a rate In of the rate that would be obtained by a second randomly selected reference laboratory. An approximate (large sample) confidence interval for the ratio In is developed. In order to account for the more general case of scaled agreement, a weighted index of agreement is also considered.