European National OS coordinator views on reforming research assessment

On November 3rd, as part of the CoNOSC Members Meeting, CoNOSC brought together over 20 national Open Science policymakers to discuss research assessment reform: in line with the Agreement on Reforming Research Assessment and in light of the approaching launch of CoARA, The Coalition for Advancing Research Assessment.

Silvia Bottaro from the Open Science Unit at the European Commission set the scene by giving a presentation on the research assessment initiative and its history. She started by outlining the development process of the Agreement, noting that over 350 organisations had been involved in the consultation. The Agreement has been open for signatures since late September and to date, almost 400 organisations have signed the agreement. Political support has been provided and has driven this forward since Sept 2020 starting with a related Commission Communication on ERA (European Research Area), followed by Council Conclusions on ERA that proposed to improve the research assessment system later that year. One year later, European Council Conclusions on the Future Governance of the ERA and the ERA Policy Agenda 2022-24 included a priority action on reforming research assessment. In February 2022, the European Conference on Open Science issued its Paris Call on Research Assessment. In June 2022, the European Council published Conclusions on Research Assessment and Implementation of Open Science.

The common vision of the Agreement is

“Our vision is that the assessment of research, researchers and research organisations recognises the diverse outputs, practices and activities that maximise the quality and impact of research. This requires basing assessment primarily on qualitative judgement, for which peer review is central, supported by responsible use of quantitative indicators. Among other purposes, this is fundamental for: deciding which researchers to recruit, promote or reward, selecting which research proposals to fund, and identifying which research units and organisations to support.”

Signatories agree to base actions on common principles and to implement 10 commitments for change. The Coalition for Advancing Research Assessment (CoARA) will provide support to those reforming research assessment. It will be launched on 1 Dec 2022. Member organisations will be able to exchange knowledge with their peers to accelerate the advancement of research assessment reform. It had been designed using both top-down and bottom-up approaches. Ms Bottaro wrapped up the presentation by pointing out that it is vital to identify and address national barriers to reform. Some groundwork for this had been done in April in a European Commission ERA Forum consultation, and discussion continued on this at the meeting.

CoNOSC members engaged in several breakout sessions to discuss the topic further. Since the Chatham House Rules apply, the names and countries are not included in this account.

Some of the findings of these sessions include:

How countries are encouraging stakeholders to join CoARA.

It should be noted that a number of CoNOSC members are early signatories of the Agreement. Selected countries already have reforming research assessment as part of a research and innovation national law.

Countries are encouraging stakeholders to sign the Agreement and join CoARA through several strategies:

·   In some countries, campaigns are underway to stimulate research funders and organisations to sign the Agreement, including rectors conferences or university associations. Other countries have a range of leading funding agencies sign the Agreement thereby not explicitly campaigning for others to sign but leading by example. It was noted that there is often no single organisation that has the authority to stimulate signing or joining CoARA.

·   Certain countries have inserted elements of the Agreement into new Open Science policy or action plans to address numerous reforms in research assessment.

·   Some countries are talking to a range of stakeholders with the view to engage with them in the broader implementation of the reform. For example, they are organising community-building workshops to discuss national barriers, establishing working groups of DORA signatories or bringing young academics together.

·   Some countries have already set up pilots for reform.

·   Efforts are also being made to discuss and align on this topic with high-level groups such as the G7 Open Science Working Group on how to reward and incentivise Open Science in a reformed research assessment system, the UNESCO Working Group on Research Assessment and the Global Research Council.

How are you promoting change and good practice in this area, and what are your plans?

·  Disciplines have very different publishing traditions, cultures and evaluation metrics. How you communicate and engage with them must take this into consideration, including the language.

·  Since ministries do not always dictate how research assessment is conducted, it can take on a role to raise awareness of good practices on current developments in the EU or by showcasing champions who are in favour of change such as forward-thinking rectors. Countries can work with them to invite them to join CoARA or local national working groups or committees to progress this forward. On a more formal level, some are presenting the idea of CoARA and the Agreement to senior bodies responsible for Higher Education and Research in their country.

·  Funding agencies can lead by example by changing their practices, thereby becoming positive role models for research assessment reform.

·  Peers are being identified to pick up the national implementation of the Agreement. We need to build national communities and achieve national momentum to convince university leadership whilst respecting the autonomy of the institution. National conversation will help bring those who are more hesitant or in silos on board. Certain countries are providing platforms for discussion amongst European experts and key national R&D actors by hosting seminars on new research evaluation strategies or by organising national conferences or festivals on the topic of research assessment. Funds are needed by some to pursue these efforts.

·  Certain countries are involved in innovative projects (EC or national) that develop and test new research assessment indicators, e.g. https://opusproject.eu/.

·  Other countries are planning to write new practical guides for research panels.

What do you see as the main national barriers to reform?

·  Some universities are more research-intensive or are highly ranked and have done better out of the current system so may be more reluctant to change. Certain prestigious individuals or institutional leadership are standing in the way of change in some cases. They may be more cautious and need more motivation from others to move.

·  Current funding incentives for RPOs, departments or even individuals in some countries are too heavily tied to bibliometrics and still heavily depend on the Journal Impact Factor (JIF). Some doctoral theses require a certain number of articles in high IF journals.

·  It is important to overcome the fear of being the first mover by having a critical mass of organisations that are aligned and willing to experiment or adopt new practices. We need collective change. This is more important for universities than it is for funders. We need to work together rather than against each other.

·  Competition between organisations is a key barrier to change. The prestige of research output and institutional rankings play an important role on international research and recruitment strategies. We are stronger in numbers through a collective effort.

·  We must be sure not to diminish the quality of science or scholarship through this change and good champions from senior management can help build confidence here. Differences between disciplines prevail and some disciplines may be less willing to transform. 

·  There is some concern that the increased need for transparency in evaluation processes may be somewhat utopian.

·  We need to translate the principles of CoARA into our national frameworks and then into regional and institutional ones and commit the appropriate resources.

·     Some countries are internally very decentralised, so a centralised approach on this topic may be challenging.

·  There are limited resources to implement change since allocating budget for research assessment means takes money from somebody else and workloads are already stretched. There might be a fear that using qualitative information in evaluation instead of quantitative information might be more time-consuming although a recent study showed that this is not the case. This type of information could be shared to try to remove or reduce that fear.

·  There is a lack of high-quality peer reviewers.

·  Transformation will take time, and we are in a transition phase. What standards (old vs. new) do Early Career Researchers align themselves with during this transition phase? 

How could you provide further guidance and support to stakeholders?

·  Encourage champions to mobilise others. Bring colleagues and structures involved in research integrity and DORA together with CoARA signatories for mutual learning exercises. Research assessment reform crosses over between research integrity and open research and these groups should work more closely together for research assessment reform.

·  Set up national chapters of stakeholders, hold workshops and other events.

·  CoARA working groups will be important to share innovative approaches with similar peers on the topic, also amongst funders.

·  Funding from the EC would facilitate change.