Exploring biases and potential effects of S&T indicators in peripheral spaces

Jordi Molas-Gallart^1*, Ismael Rafols^1,2+, Diego Chavarro² and Richard Woolley¹

¹Ingenio (CSIC-UPV), Universitat Politècnica de València, València, Spain

² SPRU (Science Policy Research Unit), University of Sussex, Brighton, England,

* jormoga@ingenio.upv.es,

This paper aims to explore the problems that emerge when S&T indicators are used in peripheral contexts, that is, in geographical or social spaces that are somehow marginal to (or marginalised by) the centres of scientific activity. In these situations evaluators and decision-makers are likely to use indicators that were designed to reflect variables relevant in the dominant social and geographical contexts --i.e. in the hegemonic countries, languages, gender, disciplines, etc.--, but that are usually not adequate in peripheral contexts.

We will examine various dimensions of periphery. First, the geographical: e.g. global south vs. global north, regions vs. metropolises (Aguado et al. 2014). Second, the social group dimension: women, the disenfranchised, the poor, or perhaps the elderly have social needs that are different from those of richer or more powerful groups --and the problems affecting the former tend be less researched than those of the later (Stirling, 2014). Third, the cognitive dimension: areas of research, such as epidemiology or surgery, that capture less attention in terms of publications or citations (and resources) than the more prestigious disciplines, such as molecular biology (van Eck et al, 2013).

This study investigates the mechanisms by which performance indicators tend to be biased against  peripheral spaces. This would include for example, bias in language (van Leeuwen et al. 2011), or disciplinary/topic coverage in conventional databases (Martin et al., 2010). An interesting issue to consider is how the overlap across peripheries, i.e. how bias in language coverage has an effect on bias in disciplines or topics covered (Archambault et al., 2006; Piñeiro and Hicks, 2015).

We discuss how these biases may have a tendency to suppress scientific diversity and shift research towards a higher degree of homogeneity (Rafols et al., 2012). We discuss how the "objectification" of excellence by means of indicators may support the diffusion of mainstream modes of research at the expense of critical or unorthodox modes.

References

Aguado-López, E., Becerril-García, A., Arriola, M. L., & Martínez-Domínguez, N. D. (2014). Iberoamérica en la ciencia de corriente principal (Thomson Reuters / Scopus): Una región fragmentada. Interciencia, 39(8), 570-579.

Archambault, É., Vignola-Gagne, É., Côté, G., Lariviere, V., & Gingras, Y. (2006). Benchmarking scientific output in the social sciences and humanities: The limits of existing databases. Scientometrics, 68(3), 329-342.

Martin, B. R., Tang, P., Morgan, M., & al. (2010). Towards a Bibliometric Database for the Social Sciences and Humanities – A European Scoping Project (A report for DFG, ESRC, AHRC, NWO, ANR and ESF). Brighton, UK: SPRU.

Piñeiro, C. L., & Hicks, D. (2015). Reception of Spanish sociology by domestic and foreign audiences differs and has consequences for evaluation. Research Evaluation, 24(1), 78-89.

Rafols, I., Leydesdorff, L., O’Hare, A., Nightingale, P., Stirling, A., 2012. How journal rankings can suppress interdisciplinarity. The case of innovation studies and business and management. Research  Policy 41, 1262–1282.

Stirling, Andy. Towards Innovation Democracy? Participation, Responsibility and Precaution in Innovation Governance. No. 2014-24. SPRU-Science and Technology Policy Research, University of Sussex, 2014.

Van Eck, N. J., Waltman, L., van Raan, A. F. J., Klautz, R. J. M., & Peul, W. C. (2013). Citation Analysis May Severely Underestimate the Impact of Clinical Research as Compared to Basic Research. PLoS ONE, 8(4), e62395. doi:10.1371/journal.pone.0062395