Criteria for assessing research quality in the humanities: a Delphi study among scholars of English literature, German literature and art history

How to assess research quality in the humanities is an intricate question. Despite the recent efforts of many initiatives, the measurement and assessment of research quality still faces strong opposition from humanities scholars, indicating that currently used evaluation schemes and tools are not tailored well enough to humanities disciplines. We have collected quality criteria from scholars in Switzerland and at League of European Research Universities (LERU) in the three disciplines, German literature studies, English literature studies and art history with a multi-round Delphi survey. The first Delphi round resulted in a comprehensive list of 19 criteria of good research specified by 70 aspects. Although 10 of these criteria are well known and commonly used in various evaluation schemes, nine criteria are not, or at least not frequently, employed in evaluation schemes. In the second Delphi round, consensual criteria and aspects (i.e. items that were clearly approved by a majority and disapproved by very few scholars) were identified in each discipline. Specifically, 11 criteria reached consensus in all three disciplines (shared criteria), six criteria were consensual in only one or two disciplines (discipline-specific criteria) and two criteria were not consensual in any discipline (i.e. ‘productivity’ and ‘relation to and impact on society’). The results of this study corroborate previous findings that researchers have not yet adopted Mode 2-related assessment criteria. Implications for research assessment are being discussed. The focus lies in particular on the mismatch of criteria between evaluators and scholars as well as on an approach to bridge such a mismatch.