One of the objectives set out in the Room for everyone’s talent (2019) position paper is ‘focusing on quality’. In the paper, we assert that the implicit and overly one-sided emphasis on traditional, quantifiable output indicators (such as number of publications, h-index and Journal Impact Factor (JIF)) is one of the causes of a heavy workload. We also note that it can upset the balance between academic fields and is inconsistent with the principles laid down in the San Francisco Declaration on Research Assessment (DORA). This was reaffirmed with the publication of the Agreement on Reforming Research Assessment (2022). The signatories of the agreement have committed themselves to abandoning inappropriate uses in research assessment of journal- and publication-based metrics, in particular the inappropriate uses of the JIF and h-index.
This is because, while bibliometric indicators tell a story, they do not tell the whole story. The traditional indicators tend to yield a one-sided picture of academic quality. In recent years, there have been numerous studies aimed at the development and use of various metrics, as well as the negative effects of their use. Those studies conclude that the advantages often fail to outweigh the disadvantages. Furthermore, these indicators are not equivalent across academic disciplines, and so fail to do justice to the diversity that exists within academic domains and academic practice. Relying too heavily on such indicators can disrupt diversity and the social impact of research, as well as impede the practice of open science. It is important, therefore, to recalibrate and broaden the assessment system for research.
To that end, we – the knowledge institutions – are striving towards a more evidence-based approach. The Strategy Evaluation Protocol 2021-2027, for instance, already assumes a qualitative assessment of research groups. A growing number of institutions – including NWO and ZonMw – have also begun working with evidence-based CVs. In this kind of CV, the academic (i.e., candidate) provides a cohesive description of their profile and achievements, supported by the most significant results (evidence) that are relevant to their specific profile, discipline and context.
An evidence-based CV certainly offers room for the quantitative underpinning of achievements. It is therefore not the case that the use of quantitative indicators will be prohibited, despite what is sometimes claimed. Quantitative indicators can still be used, provided this is done in a responsible way. The guiding principle is that academic performance and academic staff should be evaluated based on quality. We feel it is important for academics to actively engage in discourse on what constitutes good-quality research. When meaningful metrics are a valuable addition and an academic is able to explain why a certain quantitative indicator is relevant, this provides much more context and may certainly be taken into consideration in the assessment.