Expert judgement is inherently noisy, but the consequences of this variability are often underestimated. Here I examine the UK Research Excellence Framework (REF) as a case study illustrating how judgment noise can interact with structural features of a multi-stage evaluation system. Simulations show that noise in both internal departmental reviews and REF panel reviews, in conjunction with the bounded REF rating scale, produces systematic underestimates of the highest-performing departments. Panel noise, rather than internal noise, had a disproportionate impact on department rankings, with even moderate noise producing severe distortions. The case of REF demonstrates that noise embedded in sequential evaluation, for example, in hiring and grant review decisions, does not "cancel out", but instead can propagate in potentially surprising ways. At a policy level, these results underscore the need for a noise audit of REF panel ratings.
Robert Ward (Mon,) studied this question.