image from The Value of Bibliometrics, March 2011 / Matthew Richardson The Research Excellence Framework: revisiting the RAE

As part of my PhD I’m considering issues to do with intangible impact and how and why this should be measured in museums. Most, if not all, measurements of impact, of pretty much everything, concentrates on the economic dimension. And really why wouldn’t it? Measuring the quantifiable makes sense. But what I’m really interested in is intangible impact. Values that aren’t concrete, tangible or physically discernible. I’m probably quite stupid for looking at things you can’t actually quantify.

In the past museums and impact has been more to do with evaluating services in terms of outcomes, whether that be economic or social value and all really for box ticking to ensure future funding. But how does that filter down to digital technology in museums? Can you measure the subjective experiences of visitors? How do you attempt to address the notion of impact of digital technology in museums? It’s easy to think that what is trying to be identified is an already defined concept. But really, impact evaluation is complex, not helped by the fact that definitions are still being determined and understood.

So how do you measure the immeasurable?

For a while I have been thinking about something my supervisor mentioned in a meeting, about how UCL is trying to work with the forthcoming Research Excellence Framework (REF) and how they are thinking about measuring intangible evidence for impact.

Is it possible to look at the intangible impact of museums and digital technology through the lens of academic research impact?

UCL and every other HE institution are currently addressing how to capture research impacts in anticipation of the forthcoming REF. The REF now includes an impact measurement and has been met with some anxiety and unease,particularly by Humanities academics. There are concerns within the arts and humanities about the ability and the value of metrics as indicators of impact. Mostly because in the humanities, impact is more likely to be intangible than tangible.
So for the purposes of the REF, HEFCE have defined impact as:

an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia.

Impact includes, but is not limited to, an effect on, change or benefit to:
• the activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding
• of an audience, beneficiary, community, constituency, organisation or individuals
• in any geographic location whether locally, regionally, nationally or internationally.

It’s makes really interesting reading, particularly as Case Studies are required and that impact will be assessed in terms of their ‘reach and significance’ regardless of the geographic location in which they occurred. But if I’m honest the REF seems to go back on itself somewhat when it starts to ask for key outputs from the research, and evidence about the quality of the research. So despite trying to address intangible impact, academics have to resort to physical countable outputs.

Is this something museum and impact researchers should be doing, producing case studies about the intangible benefit and then trying to quantify it with additional references?

Regardless, it’s going to be really interesting to see how arts and humanities departments go about quantifying intangible impact. Maybe it’s something to look at in the future. But currently, I’m a bit stumped.