Research evaluation systems – national-level evaluation procedures for the systematic periodic evaluation of university research in a country – are governance tools that have been invented three decades ago, and have been enjoying increasing popularity in higher education policy communities ever since. They can be used to legitimise the allocation of funding, to save money, or to provide universities with both the incentives and the information for improving their research.
The assumption that research evaluation systems can indeed make universities improve their research can be doubted, however, for a variety of reasons. First, research evaluation systems rarely use valid measures of research performance. In most cases, volume rather than performance is measured. Therefore, research evaluation systems rarely provide useful information to universities and are likely to cause goal displacement. Second, the systems target organisations or organisational units rather than units actually conducting research, which opens the response to evaluations for a host of additional organisational influences including internal competition, power structures, necessities of teaching, and others. Third, evaluation systems are just one tool in a whole arsenal that is used for the governance of research. The overlap of the numerous tools aimed at governing research weakens the influence of each of them. Fourth, any influence of research evaluation systems on research performance need to ‘pass through’ the researchers, for whom they are likely to pale to insignificance compared to expectations of their scientific communities and necessities to continuously fund research.
For these reasons, universities often have only very few opportunities to shape their research.
They can redistribute time, resources and managerial support from teaching to research and from weaker to stronger researchers, they can change their recruitment practices, and they can increase the volume of research that is favoured by current evaluation procedures. Each of these measures comes at a price, though.
Interestingly enough, measuring the impact of research evaluation systems turns out to be close to impossible for the very same reasons that limit university responses. Causal attribution of effects could be achieved either by statistical association or by identifying the social mechanisms that translate research evaluation systems into improved performance. In both cases, research performance needs to be validly measured, and the influence of research evaluation systems needs to be distinguished from the influence of other factors in the long and overlapping causal chains. The latter can be achieved by in-depth studies at the micro-level but results of those are difficult to aggregate. This is why the political and scholarly discussions on the effects of research evaluation systems rarely use reliable evidence.
Discussant: Jordi Molas