‘Let’s put a stop to the opportunistic use of rankings’

The use of global university rankings is at odds with the Recognition & Rewards programme. That is the conclusion of an expert group in its recommendation paper entitled Ranking the University, which was published shortly before the summer of 2023 and was prepared at the request of the Board of umbrella organisation Universities of the Netherlands (UNL). These league tables largely base the research performance of universities on the number of publications and citations, while Dutch universities, through the Recognition & Rewards programme, aim to more broadly recognise and reward the work of academic staff and put greater emphasis on quality. As result of these findings of the expert group, Dutch universities will therefore be taking a more critical stance on the use of global university rankings.

The expert group was asked to provide advice on, and propose solutions to, issues associated with university rankings in relation to the Recognition & Rewards programme. Ludo Waltman, professor of Quantitative Science Studies and deputy director of the Centre for Science and Technology Studies (CWTS) at Leiden University, was a member of the expert group. We spoke with him about the recommendation paper.

There is a host of rankings, but in its recommendation paper, the expert group focuses solely on what are known as ‘global university rankings’. So what rankings are we exactly talking about?

“We’re talking about rankings that claim to be able to sum up a university’s performance in the broadest sense in a single score, while lumping together the university’s various activities (including education and research) and disciplines. The best-known examples are the Times Higher Education World University Rankings, the QS World University Rankings and the Academic Ranking of World Universities (ARWU), also known as the Shanghai Ranking.”

How are these university rankings actually created?

“Rankings combine data from various sources, and data concerning publications and citations play an important part here. Surveys are also used, and universities are asked to supply data themselves as well, for example on the ratio of students to staff.”

What is the underlying goal of these rankings or, to put it differently, who are they intended for?

“It seems that the main target group is made up of students (and their parents) researching which foreign university to study at. Rankings create the impression of being able to help students to find a good university to join. However, this is highly debatable, because the rankings have far more to do with research than education.”

In the summer of 2022, the Board of Universities of the Netherlands (UNL) set up an expert group and asked it to provide advice on, and propose solutions to, issues associated with global rankings in relation to the Recognition & Rewards programme. The group’s recommendation paper, entitled Ranking the University, was published in July 2023 and has received an administrative response from UNL.

University context in which rankings are used

The expert group concludes in the recommendation paper that the use of league tables is problematic. Why is that?

“League tables are misleading. They make strong claims about the information they provide, for example by saying they showcase the best universities in the world, but they are completely incapable of delivering on those claims.”

The use of league tables is at odds with the basic principles of the Recognition & Rewards programme. What exactly are the issues the expert group is raising?

“League tables are largely determined on the basis of publications and citations, whereas a heavy emphasis on publications and citations is precisely what the Recognition & Rewards programme wants to move away from. This is a significant area of tension. After all, universities aiming for good scores in league tables could easily be tempted to judge their researchers mainly on publications and citations, thus seriously undermining the Recognition & Rewards programme.”

What effect is the Recognition & Rewards programme having on the position of Dutch universities in the international rankings?

“It’s difficult to say with certainty, but I expect it to have a negligible effect. If it ends up having any effect at all, it will in all probability be far less than the effect of all kinds of external factors over which Dutch universities have little influence anyway, such as, for example, the strong growth of the Chinese academic system.”

The report makes constant reference to ‘culture change’. What specifically do you think needs to change? Is it mainly the way we want to measure the performance of universities or rather the value we attach to the rankings?

“The main change in culture will involve having to be far more honest about rankings. Some of the information that rankings provide can be fairly relevant in certain situations, but at the same time, we are all perfectly aware that rankings simply cannot deliver on their simplistic claims about what the world’s best universities are. We need to stop treating rankings opportunistically and stop accepting the problematic narrative of rankings in a wholly uncritical manner. Instead, we must dictate the narrative, by actively promoting what we ourselves see as the strengths of our universities.”

Utrecht University recently decided to stop supplying data for the Times Higher Education global ranking. As a result, Utrecht University was not included in their World University Rankings 2024. In a response, the university has stated that rankings put too much emphasis on scoring and competition, and the university actually wants to focus on cooperation and open science. Utrecht University is the first Dutch university not to participate in this global ranking. In taking that decision, Utrecht University was directly acting on the recommendations of the expert group, which advised Dutch universities to stop supplying data to non-transparent league tables in the long term – if there is international support for this.

The matrix above summarises the proposed culture change strategy with regard to league tables. The rows of the matrix show the three levels at which a change in culture should be promoted. The columns show the four areas.

In general, what does this advice mean for academics, at any of our institutions?

“By adopting our recommendations, universities will show their staff that they are unequivocally opting for the new Recognition & Rewards programme. Staff will be able to rely on universities embracing the Recognition & Rewards programme not just in word but also in deed.”

Rankings are often used for marketing. What role do you think communications advisers can play in bringing about a change in culture with regards to international rankings?

“The work of communications advisers is about to become much more interesting. Instead of blindly following rankings and framing the rankings’ results as positively as possible, communications advisers will be able to start focusing on conveying the real story of their institution. What is the institution really proud of? What sets the institution apart from other institutions? And what does this mean in practice for the institution’s education and research? These are the questions that really matter. In answering such questions, communications advisers will be making a crucial contribution to a change in culture with regards to rankings.”

Ludo Waltman was a member of the rankings expert group set up by the UNL Board in the spring of 2022. He is professor of Quantitative Science Studies and deputy director of the Centre for Science and Technology Studies (CWTS) at Leiden University. His work focuses on infrastructures, algorithms and tools to support research assessment, science policy and scholarly communication. Waltman is also coordinator of the CWTS Leiden Ranking, a ranking that stands for a multidimensional perspective on university performance.

Ludo Waltman