Public Engagement & the REF: A process of judgement, not measurement…

by Caroline Gillett, Public Engagement with Research Officer.

I recently attended a NCCPE event which addressed the relevant questions in the current HECFE consultation on the future of the REF, in the light of what we have learned from REF 2014. In a true act of public engagement, the event primarily offered an opportunity for the NCCPE to present their draft response to the sector for feedback and discussion. The workshop also featured a particularly useful Q&A session with Steven Hill (Head of Research Policy, HECFE).

steven-hill

This blog post aims to capture my own key ‘take-home’ messages. Information here is not presented as fact or verbatim, and no decisions have as yet been made by HEFCE on REF 2021 guidelines. This post also assumes you have some knowledge of the REF 2014 process. If not, feel free to drop in to a Breakfast Brainstorm session and I’d be happy to go over the basics with you if interested.

The HEFCE Consultation poses a number of questions which have particular relevance to / implications for public engagement in the REF process. Namely:

  • Question 23: What do you think about having further guidance for public engagement impacts and what do you think would be helpful?
  • Question 20: What comments do you have on the recommendation to broaden and deepen the definition of impact?
  • Question 21: Do you agree with the proposal for the funding bodies and Research Councils UK to align their definitions of academic and wider impact? If yes, what comments do you have on the proposed definitions?
  • Question 29: What comments do you have on the inclusion of examples of impact arising from research activity and bodies of work as well as from specific research outputs?
  • Question 15: What are your comments on better supporting collaboration between academia and organisations beyond higher education in REF 2021?
  • Question 35: Do you have any comment on the ways in which the environment element can give more recognition to universities’ collaboration beyond higher education?

Discussions clustered around four themes which I’ve attempted to broadly summarize below:

Enhancing public engagement guidance

  • Clearer guidelines (possibly with explicit examples) are needed for public engagement in REF2021, as existing guidelines were interpreted too conservatively in turn contributing to a skew in the direction of dissemination.
  • The bar will be raised, even if criteria remain the same. Ambition needs to be amplified as emulation of a 4* public engagement from REF2014 is no guarantee of success this time around.
  • What does instrumental, conceptual and capacity-building public engagement look like across all four panels? Inherent challenges across panels need to be acknowledged, as capacity to move beyond dissemination is understandably more difficult in theoretical/technical fields.
  • Guidelines should not be overly prescriptive to avoid copy-cat / roadmap behaviour.
  • Minimum threshold for criteria (across the star rating system), as opposed to examples, might be a means enhancing clarity without limiting diverse inputs and constraining ambition [Wellcome Trust supposedly uses an approach similar to this for its grants].
  • Ultimately guidelines need to give confidence to the sector to encourage and legitimize public engagement impact case study submissions.

Broadening & deepening the paradigm of impact

  • More disseminative forms of public engagement still have a place, however there is a need to ensure that any such activity is at the very cutting edge of research.
  • Addressing misconceptions in overlooked publics was also brought up, as there remains a tendency to default to schools-based work without a rationale for why other publics could not be equally engaged.
  • Powerful partners with unique expertise and access to other publics of interest might be a means of achieving engagement with overlooked groups e.g. engagement with lifelong learners through the Woman’s Association was mentioned.
  • The impact of research on research (which might allow co-produced outputs to be included) is still unlikely to be adopted this time around…
  • Reach and significance are intended to be measured holistically [Panel C offered particularly useful guidance here]. However, “layering dissemination” (broadening reach through follow-on websites, apps, downloads) could be used as second stage activity to augment a core, highly significant but low reach activity with deeply engaged and carefully evaluated groups over a period of time. This might make the difference between 3 and 4*. Dissemination alone however was deemed unlikely to fly.

Underpinning research

  • Thus far there has been “warm support” for the notion of bodies of work.
  • This could decouple public engagement from outputs which would open up public engagement not currently eligible, however the definition of what constitutes a body of work must be carefully considered to not bias some forms of engagement over others.
  • A ‘symptom’ of good collaboration is that often outputs are produced off the back of that collaboration. Might “associated research” as opposed to “underpinning research” be better? Yes possibly, but the notion of associated research appears to have already been rejected by HEFCE [despite support for this from Hill himself]…
  • Clearly there must be a negotiation around activity linked to outputs and activity linked to what collaborators and intermediaries (teachers, museums etc.) need and want.
  • Panels need to be sensitive to such compromises, but also question the rationale and aims for working with particular groups more closely to begin with.

Encouraging better collaboration

  • Collaboration needs to be embraced at the level of units of assessment (UOA) and articulated as such within impact templates. However…
  • Impact templates (which currently sit alongside impact case studies) are likely to be scrapped in favour of an environment section which could feasibly see “specifically earmarked” sections (e.g. public engagement) with “specific weighting” attached to them.
  • This presents a particular challenge for universities where the track record of public engagement is dependent on individuals rather than embedded within an institution/UOA strategy and practice.
  • The Earth Sciences impact template from University of East Anglia (UEA) was lauded as a particularly striking example of how public engagement was embedded and recognized within a UOA.
  • The NCCPE pointed towards three documents they felt particularly useful for thinking through collaboration:
  • Looking even further into the future, REF 2027 would likely feature some mechanism to link processes together more effectively to allow some reflective comparison of environment statements in future. If true, this should work in favour of public engagement culture change, putting pressure on universities to work towards and sustain ‘embedded’ status on the EDGE tool.
  • The NCCPE are suggesting the term ‘knowledge exchange’ (used by HEIF) might be more useful way of framing public engagement in the REF. However, there was significant resistance to this notion in the room. The concern being that this took the ‘public’ out of public engagement and endangered efforts to see that engagement is not blinkered towards industrial/business engagement at the expense of everything else.

Conclusionpaul-manners

  • Panels were lenient around evidence gathering last time around (because impact was ‘new’) but they will be much stricter next time.
  • Credibility of evidence will be strengthened on the basis of how atypical a response/testimonial is seen to be against all existing/possible responses.
  • It is anticipated that expectations will be raised towards greater level of external critical perspective and/or professional evaluation.

Ultimately however, REF remains a process of judgement, not measurement. It is a myth to assume that you can evidence impact without a compelling story. Knowing what story you want to tell can seem difficult, particularly as impact has an interesting habit of taking you places you might never have envisioned at the very beginning of a project. That’s normal, that’s expected. Impact is not a linear process and no one expects it to be. Transparency around aims, audiences and rationale in a public engagement context are key though. Yet many still become transfixed on the activity itself as a starting point. Instead, looking for the external context to motivate activity is where your public engagement story† might begin to take more meaningful shape…

/ End /

[Written with thanks to the NCCPE for a useful event. The NCCPE welcome your reactions to their current draft response. Contact nccpe.enquiries@uwe.ac.uk by March 10th, 2017 to input. For information on public engagement in REF2014, you may find my previous blog post useful, or better yet read the newly launched NCCPE report which goes into much more detail]

Additional content

General comments:

  • It is likely that there will be a word limit rather than a page limit for impact case studies in REF2021
  • Attachments in the form of images and audio/visual recordings etc. designed to complement the written text might be possible, though care should be taken to not over rely on any such material as panel assessors will have limited time to review content
  • Wellcome Trust are likely to be invited to once again have a representative on Panel A. It seems likely that this person would have an understanding of public engagement given Wellcome’s strong commitments in this area.
  • There was a sense that submission of new institutional impact case studies might not be worth the effort. The implications for interdisciplinary research remains unclear.

From the Q&A:

Could a case for the economic impact of public engagement activity (e.g. media based work) be turned in to an impact case study?

Yes. However, Hill remarked that although this was possible, the question remained as to whether this was desirable. The suggestion here being perhaps that telling a story in which public engagement was primarily motivated by the prospects of economic return would be challenging and/or odd in the very least.

Would members of the general public join the panels?

No. Hill provided a very reasonable response to this question stating:

  • To what extent can any one individual member of the public be representative of the public at large?
  • There is always a danger that such an individual would only become ‘institutionalized’ to the REF process anyway

Can there be a public engagement expert on each sub-panel/main panel?

Possible, but unlikely. This would open the door to requests for experts in a variety of others areas (e.g. tech transfer specialists) and panels would simply be unable to accommodate those with expertise in each area.

There was a suggestion that sub-panels might have members who also meet outside of each group as an interdisciplinary group to calibrate activity. Calibration of impact at panel level had proven useful last time around and this could feasibly include some calibration of public engagement activity specifically in future.

Would the NCCPE be invited to join the panels?

Possibly. However, it was more likely that the NCCPE would be invited to run training sessions for panel members in the lead up to assessment. This was the case last time.

*Background to the consultation:

HEFCE have launched a consultation to inform REF2021. The questions within this relate to the recommendations put forward by the review of REF2014 carried out by Lord Stern.

The Stern Review issued recommendations including…

“Recommendation 7: Guidance on the REF should make it clear that impact case studies should not be narrowly interpreted, need not solely focus on socioeconomic impacts but should also include impact on government policy, on public engagement and understanding, on cultural life, on academic impacts outside the field, and impacts on teaching.”

† NCCPE has found that most public engagement case studies fell into one (or some blend) of these three storylines:

  • Enlightenment & Empathy: Making the research “meaningful & persuasive” through inspiring & provoking challenge [Conceptual]
  • Social Innovation: Making the research “practical & relevant” through innovation and promoting reflexivity [Instrumental]
  • Social action: Making the research “motivating and useful” through buildings skills and creating networks [Capacity building]