Made to Measure: how measurement can improve social interventions
This piece is a companion to John Burgoyne’s blog on “Measurement for Learning”. Find that here.
My recent work has revolved around exploring alternative approaches to Public Management. Since our alternative approach does not use targets to manage performance, I often get asked the question: “does that mean measurement isn’t important?”
In this article, I will attempt to answer that question.
Measurement IS important
First things first: measurement is important — because it is a crucial part of performance improvement. We can’t get better without reflecting on what we do, how we do it, and the relationship between those two things and the effects they create in the world.
There are all sorts of things that might be useful to measure our work: we can measure elements of the processes we undertake (how many, how quickly); we can measure outputs (how many things those processes produce); and we can measure outcomes — effects in the world of those processes (client satisfaction, wellbeing etc). Some of these will be leading (“real-time”) measures, some will be lagging (only measured after the fact).
All of these are potentially useful, in different circumstances. There is no one type of measure that is simply better than another. And any of them can be done well, or poorly.
However, in the social change world we frequently get muddled in our conversations about measurement, and I think there are two key underlying reasons why this happens.
Measurable change is not the same as “real” change
Sometimes, people will say: “we want this work to create ‘measurable change’”. When people say this, they seem to be using “measurable” as a synonym for “real”.
But “measurable” change is not the same as “real” change. In social change work, real change is experienced in someone’s life. Real change is how someone’s life is different. This can be change in how they see the world, or change in their feelings. Real change is as much about these intangible things as it as about the tangible: change in behaviour, processes or structures. Real change is as much someone finding forgiveness as it is them finding a job.
All of these things (tangible and intangible) are measurable (with different degrees of difficulty and effort). But the measures are an abstraction. They are a simplification of the complex, multifaceted nature of real life into a data point. The measures are a pauperised, context-free, superficial substitute for reality.
In the same way that “the map is not the territory”, the measures are not reality. Understood in this way, “measurable change” is the very opposite of “real change”. Real change is change in the world. Measurable change is change in that tiny aspect of the world we were able to record, with those limited tools.
Measure to learn and improve, not for accountability
Again, none of this says that measurement isn’t important. But recognising the limitations of what measures can tell us helps us to use them appropriately.
The abstraction involved in measurement can be useful. Abstracting from the immediate can help address cognitive biases. Abstraction aids processes of comparison — how does this situation compare to another? These are all very useful functions.
To make good use of these functions, we must treat measures as tools to help our reflection processes. In other words, we can use measures to help us learn and improve. When we reflect on our practice, we can’t simply rely on recollecting our experience of our work because we know that they will be subject to cognitive biases — we will likely place undue importance on the most recent events, and we may select those recollections which best fit our pre-existing worldview. Measures help us to learn.
But to help us learn, measures must be part of a toolkit. What we can measure must be balanced with other types of data. We need to make sense of all the available data — the narrated experiences of ourselves and others, our feelings and impressions, as well as measures — in a process of judgement.
The key question: why are we measuring?
But to use measures to learn, we must answer the question: “why are we measuring?” We have a choice about how we answer this question, and it comes down to this: are we measuring to learn and improve, or are we measuring to be accountable to others (to demonstrate something to them)?
This choice is highlighted in Campbell’s Law:
“The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”
(A very similar idea is also expressed in Goodhart’s Law):
The choice we make fundamentally changes the meaning of the measurement processes. To illustrate Campbell’s Law in action, let us look at the Wellbeing Star. The Wellbeing Star is often used to track the progress of people who are receiving interventions designed to increase their wellbeing. It asks either the people being served or the workers supporting them (or sometimes both) to measure over time how they feel about key aspects of their lives, such as how they are managing the symptoms of long-term conditions, and how they are managing their money. It says to them: give a score out of 5 for how good you feel about this aspect today/this week etc.
One of the key uses of the Wellbeing Star is as a therapeutic tool. It acts as a record over important aspects of how a person is feeling, which they, and the person supporting them, can use as a reflection tool. Reflecting together helps the person and their worker to learn about their own wellbeing. For example, it enables them to see patterns, such as: do you tend to feel better when you see this person? Or go to this place?
Used for this purpose, the Wellbeing Star can be introduced to the therapeutic process whenever it is appropriate. Some may not be ready to score aspects of their lives early in their relationship with their worker. Or, they may not trust the worker enough in the early days of that relationship to give honest answers.
Alternatively, the Wellbeing Star can act as a Performance Management tool for teams/organisations. In my previous research, I have seen the Star used as a mechanism for Payment by Results. In this context, the measurement process is seeking to know: has your organisation increased the wellbeing scores of these people over X period of time? If so, your organisation will receive £XX. If not, your organisation does not get paid..
In this context, the Wellbeing Star had to be completed within a specified time of “first contact” between the person needing help and the organisation helping them — because the purpose of the measurement was to create baseline information from which the organisation’s performance could be assessed. And then it had to be completed again at a pre-specified time in the future, in order to assess progress within a standardised payment schedule period. The workers had no discretion about whether undertaking this scoring exercise at this time was useful or appropriate for the person being helped. And further, because the workers knew that the scores being produced directly impacted on the financial health of their organisation, pressure was created to make sure that there was a gap between the starting score and the end scores, so that the organisation would get paid.
This example helps to illustrate why the answer to the question: why measuring is fundamental, and why we have to choose. We cannot use measures for both learning and for accountability. Because as soon as we use measures for accountability, Campbell’s Law kicks in, and the effort we put into the measurement process has been wasted — because the measurement has corrupted the process it is intended to measure.
Examples of measurement for learning in practice
Local authorities across the UK are experimenting with measurement for learning and improvement, just as the Wellbeing Star can be used as a therapeutic tool, not a performance management tool. Follow the Upstream Collaborative’s meaningful measurement working group to see the values and principles that underpin this approach and more examples of what it looks like in practice.
And if you’d like to see further case study examples of measurement for learning in practice, you can find a whole raft of case studies and tools at www.humanlearning.systems.
Measurement and accountability
All this raises one final question: if we know that we cannot use measurement for accountability, how will accountability work? We’ve started to explore this question here. The key to resolving it will be to treat it as a collective problem to experiment with: who needs to hold whom to account, and in what contexts?
For example, we want the media to hold our politicians and public servants to account. How can they do that in a way which promotes a culture of learning, rather than hindering it? This isn’t a question that can be answered in the abstract. This can only be achieved by politicians, public servants, citizens and journalists working together to create new forms of accountability. Let this work begin now.
