A group of 175 scientists, including Nobel laureate Ben Feringa, are deeply concerned about the new assessment method that Utrecht University will introduce, among others. The university no longer wants to judge researchers based on the so-called impact factor, a score that indicates how much a researcher has published and which scholarly journals end up with these publications. Because of this impact factor, much emphasis is placed on the scholar’s research publications and little attention is paid to other qualities, such as leadership. The group of 175 people believe that the university’s decision is wrong, because this influence factor is critical to their international career. Four questions about the evaluation of scientists.
Who evaluates scientists and how?
Scholars evaluate each other when new colleagues are appointed or in the event of a potential promotion. For this assessment they use, among other things, the impact factor. The more colleagues who refer to your business, the higher the impact factor for that business. Publications in frequently cited journals such as nature On Science Higher scores than publications in less cited journals.
Many organizations, such as the research funder NWO and the University of Utrecht, would like to conduct the assessment of scientists in a different way. What will they do and why?
From the beginning of 2022, the University of Utrecht will no longer include the impact factor in the recruitment and evaluation of scholars. Instead, the university will evaluate employees based on a new system called recognition and value. This is the result of a national initiative involving all universities, UMCs and organizations such as research funders NWO and ZonMw. The focus here is on social impact, collaboration, and open access to science.
Paul Boselli, chair of the Utrecht University Recognition and Recognition Committee, says at a interview in nature The influence factor contributes to the “productivity” of science: the focus is on production, not on the quality of the individual world.
Research funders NWO and ZonMw are also adjusting their criteria for evaluating new proposals. Jeroen Gorts, a brain researcher at the Amsterdam UMC and director of the NWO and ZonMw, stated that scientists should always have a ‘five-legged sheep’ as an important reason for the new rating system. Researchers must excel in research, but also in education, patient care and management, for example. Many scientists suffer from a very one-sided focus on research performance, as a result of which other tasks are not valued. The new system thus gives scholars the opportunity to present themselves and to choose the areas in which they excel and in which they want to be evaluated.
What is the criticism of the new scientists’ assessment?
This week I published a group of 175 scientists Opinion article in ScienceGuide, a platform for news from higher education and science, where they express their criticism of the new way of evaluating scientists.
The authors argue that policy harms the quality of science because new standards are difficult to measure. In the international context, the impact factor is an important indicator, and by not including it in the assessment, young researchers in particular will be the victims, as they can no longer keep pace with international competition.
By the way, not all young researchers agree with this, says Hanneke Hulst, a brain scientist at the Amsterdam UMC. She argues that young researchers are eagerly awaiting new recognition and evaluation. You can also evaluate high-quality science using a different system. We need to look at research talent in a different way and not use a number generated by calculating achievements that says little about an individual’s talent.” Holst contributed to a The piece is signed by 113 young scientists, in response to an opinion piece by 175 scientists.
Raymond Putt, a biochemist at Erasmus MC Rotterdam and one of the authors of the opinion piece, says the impact factor can be very valuable for some areas. The impact factor isn’t perfect, but it’s not without value either. Moreover, it is one part within a broader palette of indicators on which you rate scientists. By eliminating this, you are replacing a measurable system with one that cannot be measured. How will you measure leadership?
Although the controversy now appears polarized, it remains to be seen if opinions diverge, according to both Poot and Geurts. Butt: ‘We’ve never said that the impact factor has to be all expendable, but we think banning it completely is very drastic. One option might be to differentiate assessment in the different sciences, for example by including an impact factor in the exact sciences but not or less in the social sciences.
NWO’s Geurts and ZonMw think it’s a good idea to start the discussion. “It makes sense for cultural change to come more often. We are in the middle of a conversation where we give the gist of appreciation. Together we will define what the new assessment structure will look like.
How did the new classification system originate?
Utrecht University’s new assessment system “recognition and value” does not stand alone, but is part of an international movement towards a different way of evaluating scholars. The decision came in Utrecht to sign the Declaration on Research Evaluation (DORA). The statement was created in 2012 and the university signed this statement two years ago. 20,000 individuals and organizations have so far signed on to DORA, with the goal of improving the assessment of scientists and their publications by banning influence factors.