An evaluation of metrics used by the Performance-based Research Fund process in New Zealand

The New Zealand Performance-based Research Fund applies a set of metrics to assess researchers and rank disciplines and universities. The process involves giving a total weighted score (based on three components) to individuals and then assigning them to one of four Quality Categories (QCs), used to derive Average Quality Scores (AQS). This paper evaluates the properties of these metrics and argues that QC thresholds influence the final distribution of scores. The paper also demonstrates that the derivation of AQSs depends on the weights assigned to each QC and the distribution of portfolios. The method used to determine raw scores also has an independent effect on the distribution of scores. The paper compares how research rankings of New Zealand universities would vary if alternative summary measures, based on the total weighted scores rather than QCs, were used to evaluate performance.