A Critique of Irvine and Martin's Methodology for Evaluating Big Science

The work of John Irvine and Ben Martin (IM) deserves serious attention. The range of their studies is wide, they address themselves to issues of importance, particularly to science policy makers, their output is considerable, and they have made a marked impact not only in the scientific community, but also well beyond. Yet impact is not the same thing as quality, as IM would readily agree. And though there is much to recommend in IM's work; its value is, we believe, vitiated by a number of methodological inadequacies. It is on these that we wish to focus in what follows. The paper is divided into three main sections. In the first we summarize briefly the main stages followed by IM in presenting their results. We then go on, in the core of the paper, to criticize several aspects of their methodology. In conclusion we try to assess the value which their work has, despite the limitations we have identified. Rather than trying to cover the whole corpus of IM's work, we base our discussion primarily on their recent studies on high-energy physics, and CERN in particular.2 There are two reasons for this.