The VQR, Italy's second national research assessment: Methodological failures and ranking distortions

The 2004–2010 VQR ( R esearch Q uality E valuation), completed in J uly 2013, was I taly's second national research assessment exercise. The VQR performance evaluation followed a pattern also seen in other nations, as it was based on a selected subset of products. In this work, we identify the exercise's methodological weaknesses and measure the distortions that result from them in the university performance rankings. First, we create a scenario in which we assume the efficient selection of the products to be submitted by the universities and, from this, simulate a set of rankings applying the precise VQR rating criteria. Next, we compare these “ VQR rankings” with those that would derive from the application of more‐appropriate bibliometrics. Finally, we extend the comparison to university rankings based on the entire scientific production for the period, as indexed in the Web of Science.