Key points are not available for this paper at this time.
A formula is proposed to estimate the reliability with which an examination assesses the individual merits of the candidates, using only the marks assigned naturally by the examiners. This formula, which is founded in classical test theory, can be applied (as here) to an examination in which the candidates are offered a wide choice of questions and different questions are marked by different pairs of examiners. The use of the formula is illustrated by reference to a certain university examination; and the results of analysing the correlations between marks independently awarded by different examiners are compared with previous studies of the reliability of university examinations, with similar studies of GCE examinations, and with the precision of absolute judgement of simple stimulus magnitudes.
Donald Laming (1990) studied this question.