What’s Your PER-REF-erence?

It’s mid-2018, which means that REF 2021 is looming large. If you’re a stranger to REF, or the Research Excellence Framework (lucky you!), then let me explain. There are around 162 Higher Education establishments in the UK in receipt of Government funding. Those establishments employ around 500,000 staff, including about 200,000 researchers and serve a population of around 2.3 million students, based on recent data. Different Institutions can be vastly different in all sorts of ways so central funding is allocated according to ‘excellence’, but how on earth is that possible to quantify? For one research area ‘excellent’ might be one paper every decade, but leading to a fundamental change to the way we live or understand the world many years hence. For others ‘excellent’ might be small, iterative changes to process that incrementally improve peoples lives, but whose ripples are (intentionally) almost imperceptible. Well REF tries to do just that. It takes into account the obvious things, like publications, normalises data as far as possible, like splitting subjects into defined groups, adjusting for staff numbers etc and then tries to divvy up who the most and least excellent are.

But REF also tries to take into consideration the wider research culture and effect of research on society, by assessing ‘impact’. Put simply, this is a measure of how big an effect the research has had outside of academia. Be it through engaging with the public and impacting behaviour, perceptions and attitudes or through commercialising research to make business more efficient or effective or in influencing policy to ensure that legislation is in line with the most up-to-date evidence. Now, whilst trying to fairly assess and compare over 150 fundamentally different institutions according to their publication records is very difficult, trying to also take into account and quantify all of the myriad possible effects on anyone and everyone outside of the walls of academia is nigh-on impossible. All of this means that the guidelines and process for REF are revised each time to try and make sure they’re fair and reflect current best practice. The Draft Guidelines for REF 2021 were published in July and are open for consultation until October.

So what does this mean for public engagement? Obviously public engagement is a key way in which research can have an impact on society it’s also a very broad church. This means PER can be difficult to define and in some instances can be tricky to evidence and evaluate in the same way as other impacts like, for example, business agreements. So it’s important that the REF Guidance around Public Engagement works well and positively reinforces PER as an intrinsic part of research excellence. The risk is, if the guidance makes it difficult for assessors to understand or recognise all of the types of brilliant public engagement and their importance, Universities who include more innovative types of high-quality engagement will do worse than those who minimise the inclusion of PER. That in turn could discourage institutional support for PER and diminish its perceived value. So on 14th September the NCCPE arranged a meeting for around 100 Public Engagement and REF heads from across the UK to get together and thrash out the new guidance.

So what happened? Well, we talked, and ruminated and discussed and confabulated and got passionate (yes, really, about REF Guidelines!) because this really is a tough thing to get right. But in short there were a few key take-homes:

1. Public Engagement is conspicuously absent. Whilst the few explicit mentions of PER are positive and seemingly intentionally vague, which leaves plenty of room for the inclusion of different types of engagement, this could also leave a lot of room for variation in how engagement is recognised and assessed.
2. Annex A could try harder. Whilst the inclusion of Annex A (essentially a great, big list of different types of indicators or evidence to support impact types, including engagement) was widely praised, the NCCPE suggested that separating these indicators into three categories and then re-jigging the table accordingly, could make the assessment of impacts, especially those arising from PER, more fair and consistent.
3. Not all engagement is created equal. Just like research, some PER is easier than others, and it doesn’t always correlate that reaching the most people, or getting those most feedback means bigger impact. Working intensively with a few local people might be more impactful than going on the radio to millions, but not always perceived as such. Some audiences might take much more work than others and sometimes engagement is about getting lucky or a sudden change in what’s newsworthy. The NCCPE suggest that equality and diversity issues and the rigour taken in the approach (as well as the robustness of the evidence) should be explicitly included in the guidelines. Though the response in the room was quite mixed, it was predominantly negative and it was instead thought that the focus should be on all claims being supported by detailed and robust evidence.

The REF 2021 Guidance Consultation is open until noon on 15th October and the final guidance will be published in early 2019.