We connected with Dr Toby Lowe via our work on the construction sector transformation plan after exploring human learning systems as a mechanism for system transition. I wanted to share his article on Soapbox, but the link wouldn't show correctly on LinkedIn, so I've copied it in below.
The interesting thing for me is the dialogue of how people, who are talking fervently about systems thinking and approaches, may need to rethink a fundamental tool in their kit around measurement and management.
Confidence to proceed with system transition work does not mean certainty of outcome.
Original Clip source: #Soapbox: The sorry tale of ‘Outcome-Based Performance Management’ – Slugger O&apos
#Soapbox: The sorry tale of ‘Outcome-Based Performance Management’
Originally published on Soapbox on July 5, 2016, 4:51 pm
Dr. Toby Lowe is a Senior Research Associate at the University of Newcastle. Here he lays out in detail some of the flaws in a branded system ostensibly constructed to ensure delivery of government policy which the current NI Executive parties have agreed to implement at Stormont.
The Northern Ireland Executive has just published its Draft Programme for Government Framework. Many people will no doubt want to comment on the priorities it contains. But there is actually a more fundamental problem – the Framework is based on “an outcomes approach”.
This approach will undermine the effective delivery of any policy it contains. Northern Ireland is about to join the back of a long and dismal queue – places which are still waiting to see real improvements in the social interventions which make people’s lives better, despite have spent a huge amount of scarce resources, and wasted lots of precious time, implementing an outcomes-approach.
Let’s talk about outcomes
The Government has stated that its approach “will focus on outcomes”, and that “Outcome-based approaches have gained currency internationally in recent years”. It even references the high-priest of outcomes himself, Mark Friedman, and his well-known book “Trying hard is not enough”.
What does all this mean? At the policy level, thinking about outcomes can be a very sensible thing to do.Political activity is supposed to enable people to have a better life. Thinking about the outcomes that political activity could help create is a good tool for building a shared vision about the kind of society we want.
Turning those desired outcomes into indicators, and measuring progress against those indicators, can also be helpful for policy discussions. Using those measurements to inform rigorous debate about policy priorities is a good thing to do, as it helps us to understand which issues are most pressing, and what the trends in those issues are.
So, yes, let’s talk about outcomes. But we cannot do is seek to manage the performance of our public services, and other social interventions, using Outcome-Based Performance Management approaches.
Let’s not use outcomes to performance manage social interventions
Why shouldn’t we do that? Because the evidence is clear – Outcomes-Based Performance Management (OBPM) doesn’t work. The Executive is right, many people and organisations internationally have tried to implement them in the last few years, and here’s the summary of what they found:
“The overall conclusion from international experience of implementing an outcomes approach is that the journey is long and the results are disappointing.” (Wimbush 2011)
Every study of the impact of using OBPM says pretty much the same thing (Perrin 1998; Van Thiel and Leeuw 2002; Mayne 2007; Chamberlain et al 2010; Soss et al 2011; Keevers et al 2012; Newton 2012). Taking all this research together, we see that the good aspect of using such approaches were that they enabled people to talk about social goals together. But as soon ‘outcomes’ or ‘results’ are used to performance manage the people and organisations who seek to deliver those results, they create perverse incentives, which get in the way of people doing their jobs well.
Just this week, a systematic review of the studies of outcome-based contracting was published – a study of all the studies into the effectiveness of outcomes approaches. Want to guess what it found?
“The findings indicate that while outcome-based contracts deliver the measures of outcome for which they pay, these measures do not always reflect the intention of the contract designers, or desirable outcomes for the end-client. Measures of outcome that were not related to payment did not improve and sometimes worsened.
“Some outcome payments created incentives for service providers contrary to the achievement of desired outcomes. For example, employment services contracts that were meant to increase tailoring and flexibility had the opposite effect” (Tomkinson 2016)
Why do Outcomes-Approaches Fail?
Previously, people who have studied the implementation of OBPM have thought that such approaches fail to work because there has been some technical problem with their implementation: they weren’t measuring the right things, or they weren’t measuring them properly.
This would be good, because technical problems can be fixed. But it’s not a technical problem – OBPM fails because it’s conceptually flawed. It has two problems which cannot be fixed:
The Measurement Problem
The idea of outcomes approaches is that they improve the lives of the people who experience those interventions. Unfortunately, measuring the impact of an intervention in someone’s life is a very expensive and research intensive thing to do.
For example, Robert Schalock, who wrote the book on Outcomes-Based Evaluation (2011), says that if you want to understand the impact of an intervention on someone’s life you need to do both qualitative interviews and quantitative questionnaires (which themselves have been designed by people who have experienced the intervention), and he recommends a minimum period of 22 months research after the programme has finished.
So, people never do that. Instead, they measure what can be measured and call that an outcome. They use what are called ‘proxy indicators’ – things that are easy and quick to measure, rather than a real-life outcome in all its complexity.
And here’s the problem, if you use these proxy ‘outcomes’ as measures to hold organisations accountable to – you’re holding them accountable for what is easily measurable, not what is important.
If you hold an organisation accountable for making changes in simplified proxy indicator, the difference between that proxy and a real outcome in someone’s life is exactly a measure for the way in which that target distorts the practice of those who are being held accountable. You’re making them work to the measures, not what matters, because what matters is always too complex to fit into a simple measure.
Proxy indicators turn everyone into a caricature, and then waste time and money trying to provide inappropriate services to that caricature, rather than helping the real person in front of them. Want to see what happens when services use proxy measures to understand people? Here’s a great example…
They tell champion fitness instructors to take more exercise, and go on a calorie-controlled diet, because the proxy indicator (in this case, Body Mass Index) says they’re overweight. We all know that what matters in our own lives are too complex to be summed up in a simple measure. But somehow we think such simplification is good enough for others.
The Attribution Problem
The measurement problem is significant, but it’s not the worst flaw that bedevils OBPM. The most significant problem is that the premise which underpins OBPM – that outcomes can be attributed to particular programmes or organisations –is simply wrong.
OBPM seeks to make organisations (or people, teams or programmes) accountable for outcomes, because they need to know whether those organisations have achieved the results that they are supposed to.Should we recommission this organisation to deliver this intervention? OBPM says ‘look at the results they achieved’.
Payment by Results, which is the most extreme form of OBPM, takes this one-step further – it says that organisations should only be paid if they achieve certain results. To hold programmes or organisations accountable for producing results, you must be able to identify who has been responsible for producing which outcomes.
The trouble is, that’s impossible. Outcomes are not produced by organisations (or programmes, teams etc). Real-life outcomes are produced by a huge range of factors and interventions working together (in technical terms – outcomes are emergent properties of complex systems).
To understand this, all anyone needs to do is look at this systems map of obesity. It was produced by the Government Office for Science in 2007, and it is an amazing piece of work.
It shows all the factors which contribute to whether a person is obese, and how they interact. At a glance, we can all see that the outcome – whether someone is obese or not – is the result of a huge range of factors which are beyond the control (or influence) of any intervention, or even combination of interventions.
To hold organisations accountable for outcomes is to make them accountable for things they do not control. Even Mark Friedman acknowledges this, but his response is striking:
“Don’t accept lack of control as an excuse… If control were the overriding criteria for performance measures then there would be no performance measures at all. ” (Mark Friedman, Results-Based Accountability Implementation Guide)
Making organisations accountable for results which are beyond their control means that organisations learn instead to manage what they can control – the production of data. This is exactly what the evidence says happens , and why outcomes approaches often produce good-looking data. It’s because the organisations are concentrating on data production at the expense of actually doing the complex reality of the job that clients need.
And this is the most profound danger of all from OBPM – the data starts to tell policy makers things are improving when they’re not. If people are being made responsible for things they don’t control, and are being measured using indicators that they know don’t reflect people’s real lives, then they start to lie. This is the account provided by one staff member who has worked under OBPM:
“It [an outcomes orientated system] wouldn’t be honest. Because everybody manipulates the outcomes it just wouldn’t be honest, it wouldn’t be the same. You’d be frightened to speak because you might have put an outcome for a client that’s not a true reflection so how could you sit there in the team and say, “I feel like I’ve failed with this client,” when you’ve ticked a positive outcome.” (Lowe, Wilson and Boobis 2016)
It’s no coincidence that Outcome-Based Accountability talks about ”turning the curve” – OBPM is supposed to make the numbers look better, to make changes in the shape of the graph.But the problem is, even when it achieves this, the numbers cannot be trusted because the numbers are not real. They’ve been produced by people whose job it is to produce good-looking data.
What’s the most frustrating thing about this? We’ve known that this is exactly what happens since 1976. This is the year that Donald Campbell, an American social scientist wrote his famous essay “Assessing the impact of planned social change”. In it he formulated Campbell’s Law:
“The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”
Northern Ireland can only hope against hope that, magically, it escapes the dismal path which is laid out before it. But those taking it down that path can have no excuse that they were not warned about what lies ahead.
Comments