How do you measure the immeasurable in museums? And will the academic REF teach us anything?

image from The Value of Bibliometrics, March 2011 / Matthew Richardson The Research Excellence Framework: revisiting the RAE

As part of my PhD I’m considering issues to do with intangible impact and how and why this should be measured in museums. Most, if not all, measurements of impact, of pretty much everything, concentrates on the economic dimension. And really why wouldn’t it? Measuring the quantifiable makes sense. But what I’m really interested in is intangible impact. Values that aren’t concrete, tangible or physically discernible. I’m probably quite stupid for looking at things you can’t actually quantify.

In the past museums and impact has been more to do with evaluating services in terms of outcomes, whether that be economic or social value and all really for box ticking to ensure future funding. But how does that filter down to digital technology in museums? Can you measure the subjective experiences of visitors? How do you attempt to address the notion of impact of digital technology in museums? It’s easy to think that what is trying to be identified is an already defined concept. But really, impact evaluation is complex, not helped by the fact that definitions are still being determined and understood.

So how do you measure the immeasurable?

For a while I have been thinking about something my supervisor mentioned in a meeting, about how UCL is trying to work with the forthcoming Research Excellence Framework (REF) and how they are thinking about measuring intangible evidence for impact.

Is it possible to look at the intangible impact of museums and digital technology through the lens of academic research impact?

UCL and every other HE institution are currently addressing how to capture research impacts in anticipation of the forthcoming REF. The REF now includes an impact measurement and has been met with some anxiety and unease,particularly by Humanities academics. There are concerns within the arts and humanities about the ability and the value of metrics as indicators of impact. Mostly because in the humanities, impact is more likely to be intangible than tangible.
So for the purposes of the REF, HEFCE have defined impact as:

an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia.

Impact includes, but is not limited to, an effect on, change or benefit to:
• the activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding
• of an audience, beneficiary, community, constituency, organisation or individuals
• in any geographic location whether locally, regionally, nationally or internationally.

It’s makes really interesting reading, particularly as Case Studies are required and that impact will be assessed in terms of their ‘reach and significance’ regardless of the geographic location in which they occurred. But if I’m honest the REF seems to go back on itself somewhat when it starts to ask for key outputs from the research, and evidence about the quality of the research. So despite trying to address intangible impact, academics have to resort to physical countable outputs.

Is this something museum and impact researchers should be doing, producing case studies about the intangible benefit and then trying to quantify it with additional references?

Regardless, it’s going to be really interesting to see how arts and humanities departments go about quantifying intangible impact. Maybe it’s something to look at in the future. But currently, I’m a bit stumped.

2 thoughts on “How do you measure the immeasurable in museums? And will the academic REF teach us anything?

  1. These are really interesting questions, Clairey, and ones that I’ve thought on from time-to-time.

    A while ago, I came across some work by some researchers looking at science and innovation policies that might be worth looking into, as it seeks to use public value as a measure of success.
    Public Value Mapping: Developing a Non-Economic Model of the Social Value of Science and Innovation Policy

    Although this is not exactly the area you are looking at, there might be something applicable in their work.

  2. Hi Clairey,
    This is a really interesting post. I have been grappling with the issue of measuring impact within some research I’m doing about artists who develop participative social-media led projects. I have drawn the same conclusion as you that impact within institutions isn’t measured beyond social/economic benefit. I’ve found that for participative projects this can lead not only to box ticking but ‘synthetic’ rather than ‘natural’ outcomes as activity is often driven by an extrinsic agenda with specific targets. It has reassured me to know that you are pondering similar issues and I’m grateful for the nod to REF. Good luck with your continued quest for answers! 🙂

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s