What constitutes a high-quality researcher? Who is most suitable to dive into a particular research question? These are questions that need to be addressed by the committees judging research proposals for NWO’s Veni, Vidi, and Vici Talent Schemes. Does the narrative CV help them reach a fair allocation of the available funding? We ask Robbert Hoogstraat, project leader of Recognition & Rewards at NWO, who helped implement the narrative CV for the Talent Scheme.
“One of the main goals is to enable academic professionals to diversify their careers. In the past, those who had spent years focusing mainly on teaching, data-stewarding, or outside of academia had a very slim chance of receiving a grant in NWO’s Talent Scheme. Candidates who had put all their time and effort into research however would, in some cases, hand in a CV of 50 pages listing all their scientific publications, conference contributions and other research output. And the quantity of these and other metrics would be the deciding factor for the best candidate. This made leaving research a one-way affair because you could never compete on quantity with someone who didn’t diversify.
However, experiences outside of scientific research can also bring about valuable skills, knowledge or experience for a particular research project. For example, teaching at a primary school comes with certain insights into children’s development and the education system, or working as a journalist on drug-related crimes might come with a useful network, enabling that person to set up a unique study in that field.
Also, horizontal mobility within academia might yield valuable skills. Experience in mentorship, for example, can be valuable when applying for a Vici grant, where you will be overseeing a research project and can use your HR skills in guiding your team.
The narrative CV aims to give contenders the possibility to diversify the types of achievements they list when applying for a grant. Like in many job application procedures outside of academia, where applicants are expected to explain how their education and work experience contribute to the knowledge and skills required for a specific position. Candidates tailor their CV and motivational letter to the required skill set, instead of handing in a 50-page CV with everything they’ve ever done. This does not mean scientific experience is not relevant in the narrative CV-format. On the contrary – we are explicitly selecting the best researchers. But this format also allows for the possibility that other experiences might be valuable for the particular research project.”
“The ambition of Recognition & Rewards is to value all the different roles people fulfil in the academic context: not just research, but also teaching, outreach to the general public, and all the organisational roles, such as managing a research institute. NWO is an organisation that funds scientific research, so our possibilities for rewarding these other roles are limited. However, in our allocation of research funds, we can help to enable this transition by making sure people with a diverse range of experiences are taken into consideration for research grants, so venturing outside of scientific research is no longer a one-way street.”
“These things go hand in hand. We know quantitative assessment on full publication lists and the h-index are not always reliable indications of research qualities, as they come with malpractices such as manipulation of authorship sequence, self-citation and coercive citation. So if these practices subside, that would be wonderful. But the main goal of the narrative CV from the Rewards and Recognition point of view is to enable more diverse career paths for academics.”
“It’s not easy to say something conclusive about that as the career paths that we try to accommodate remain an exception, so gathering sufficient data for a quantitative analysis may take a while. However, based on feedback we’ve collected from researchers in the assessment committees and from applicants, we get the sense it is generally well-received. Committee members tell us they can reach a proper evaluation and get a sense of candidates’ relevant experiences when judging their aptitude for the proposed research. They also mention that coming to this evaluation takes more time than comparing h-indexes or other metrics of the different candidates. Meanwhile, we hear back from people who have received funding, and realise they would not have had a chance under the old system. So we do get the impression that the narrative CV is doing what we set out to do.”
“Obviously we cannot stop committee members from looking up candidates on Google. However, this is not in line with the briefing we send out to them, nor with the impression we get based on their feedback. Also, the data committee members might look up online won’t constitute the first input they receive about a candidate. This means the metrics will be less influential in shaping their evaluation, a phenomenon known as the anchoring effect. Meanwhile, we do also adjust our practice. We put more effort into explaining our approach and the underlying ideas to committee members and are more explicit in the types of data they should and should not take into account. We have also put personal data at the end of the form so as to avoid another manifestation of the anchoring effect.”
“In some cases this might be true. And you might argue the same for more introverted people, for example. But the problem is that the previous system – judgment based on metrics and quantity – has also proven to be a muddled indicator of quality and be most beneficial for a certain type of researcher: the type that has focused on quantity over quality. The phrases ‘publish or perish’ and ‘salami slicing’ are common ones because of the incentives given by assessments focused on metrics and quantity. Furthermore, the Swiss funder SNSF has done a content analysis on their narrative CV, which shows no significant differences between genders in the way they use it. So yes, the narrative CV might not be perfect, but so far it has shown that metrics such as the h-index or Journal Impact Factor disadvantage groups such as women and non-Western academics.
“We are looking for a balance. On the one hand, we want an open, undefined process where the applicant is free to judge which experiences are relevant for the proposed project. And on the other hand, we want to give enough of a framework for both applicants and committee members, so that we do not sacrifice comparability, and the substantiated content can make the difference.
“In the search for this balance, we swung from a very defined and one-dimensional approach to the very open and more undefined narrative CV. In the coming years, the pendulum will swing back a little bit. We are now progressing from the narrative to the evidence-based CV. This means we will keep asking for a mainly qualitative analysis, supported by quantitative and/or qualitative indicators, but we will provide more guidance in the types of information we deem responsible. For example, if one of the articles in your key outputs made a big impact in your field, you can state that it was the most cited article in that volume of the journal. That is a meaningful indicator of its relevance. But we will not return to the h-index and Journal Impact Factor.”
“We have written the proposed policy and will be sending it out to scientists from different fields to ask for their feedback. Once we have revised it, we hope to start using it in the Talent Scheme as of 2023.”
The narrative CV allows candidates to write their own story without the restrictions of prerequisite questions. It consists of two parts. The first is the academic profile, where the candidates describe what makes them the right person to execute the proposal. This profile can be substantiated by key output: a list of a maximum of 10 items, ranging from research papers to data sets, software, patents, education, etc. Open Science practices can be marked in this output list for special recognition. This set-up remains the same in the evidence-based CV, but the format will provide more structure for both applicants and committee members to clarify what is and is not allowed, and to improve the comparability between applications.
In this video Marcel Levi, President of the Dutch Research Council, explains his perspective on the Recognition & Rewards programme.