How should research performances be measures? Evidence from rankings of academic economists

Billions of euros are allocated every year to university research. Increased specialisation and international integration of research and researchers has sharply raised the need for comparisons of performance across fields, institutions and individual researchers. However, there is still no consensus regarding how such rankings should be conducted and what output measures are appropriate to use. We rank all full professors in a particular discipline, economics, in one European nation using seven established, and some of them commonly used, measures of research performance. Our examination shows both that the rank order can vary greatly across measures, and that depending on the measure used the distribution of total research out-put is valued very differently. The renowned KMS measure in economics stands out among the measures analysed here. It exhibits the weakest correlation with the others used in our study. We conclude by giving advice to funding councils and others assessing research quality on how to think about the use of both quantitative and qualitative measures of performance.