How measuring impact gets in the way of real world change
By Toby Lowe. Originally published on the Integration and Implementation Insights blog.
Why is the idea that we can measure our impact to understand how well we are performing fundamentally flawed? Why is it impossible to “demonstrate your impact” in complex environments?
Although the idea of measuring impact is seductive, almost all useful social change is achieved as part of a complex system. In other words, your work is a small part of a much larger web of entangled and interdependent activity and social forces.
The systems map of the outcome of obesity, shown in the figure below, illustrates this perfectly – it shows all the factors contributing to people being obese (or not), and all the relationships between those factors.
This is the reality of trying to make impact in the world – your actions are part of a web of relationships – most of which are beyond your control, many of which are beyond your influence, quite a few of which will be completely invisible to you.
All of these things combine with your actions to create impact in the world. Let’s work this example through using the obesity systems map. Say that you’re one of the people operating in the bottom right corner of this system – you’re providing “healthcare and treatment options” to address obesity. Let’s say you’re delivering weight loss programmes in neighbourhoods. How would you distinguish the impact of your weight loss programme from the influence of all the other factors in this system?
Short answer – you can’t. Someone on your programme sees a film that changes their perspective on the meals they cook. Someone on your programme changes jobs, to a place with a canteen where they only serve healthy options. Someone is made redundant, so they can’t afford to buy organic food. What was the impact of your programme in these situations?
This reveals a fundamental truth about the nature of complex systems. In a complex system, it is impossible to distinguish the effect of particular actors on the overall pattern. This is because complex systems produce emergent, nonlinear behaviour. The tiniest change in input variables creates potentially huge changes in results. Consequently, you can’t produce a reliable counterfactual in a complex system. (You can’t say what would have happened if X wasn’t present). And if you can’t produce a reliable counterfactual, then you cannot reliably identify the impact of your activity.
Impact isn’t “delivered”
If we want to achieve impact in the world, the crucial uncomfortable truth that must be faced (from the perspective of traditional management thinking) is that impact isn’t something that can be “delivered.” In fact, the whole “delivery” mindset is damaging to creating impact in the real world.
We have been encouraged to believe a fantasy – that we can “deliver” impact by a linear planning process. While this has appeal by making us feel more in control of the world than we actually are, looking at the actual evidence on how outcomes are made (like the systems map of obesity), we can see that this kind of programme logic model does not represent an accurate or robust portrayal of how outcomes are really made.
When we think of impact as something we can ‘deliver’, we are pretending to ourselves to make the task of managing social change easier. And the purpose of good management is not to make the task of management easier, it is to confront the uncomfortable messiness of how the world actually works. If we care about making impact in the real world, we need to stop pretending.
This kind of pretending matters because it makes the work of achieving real change in the world harder to do.
At its least worst, it wastes everybody’s time attempting the impossible – time spent “demonstrating your impact” or creating linear programme logic models is essentially time spent inventing a fantasy.
But the truly pernicious aspect of “demonstrating your impact” or linear programme planning is when it is used for accountability or governance purposes. When people, teams or organisations are rewarded for demonstrating impact – when funding is given to those who can ‘prove’ their impact, or contracts awarded on this basis, or promotions/pay rises are secured in this way – it corrupts the information we need to improve how things are working.
We know this is the case because this is what the evidence overwhelmingly and unarguably tells us happens when we seek to use “impact” (or outcomes) for accountability, governance and/or performance management purposes. The key point is summed up in Campbell’s Law: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor” (Campbell, 1979, p.85).
What can we do instead?
If we’re not asking organisations to demonstrate their impact, how can we create accountability for spending resources well?
This is a great question. Fortunately, it also has a couple of straightforward answers.
- Remember – asking people and organisations to demonstrate their impact doesn’t create accountability – it creates a bunch of fantasy data.
- Ask people and organisations to be accountable for experimenting and learning together – collaboratively. Create accountability for enabling the healthy systems, which are how positive outcomes are actually achieved in the real world, as highlighted in my previous i2Insights contribution, Managing complexity with human learning systems.
The key shift to make here is to move from funding for “demonstrable” impact (because this – paradoxically makes real impact harder to achieve) to funding for collaborative learning and adaptation. This is how real impact is made.
What has your experience been? Do these ideas resonate with you?
To find out more:
This i2Insights contribution is an extract from Lowe, T. (2023). Explode on impact. Medium, 23rd June. (Online): https://toby-89881.medium.com/explode-on-impact-cba283b908cb
References:
Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning, 2: 67-90.
UK Government’s Foresight Programme. (2007). Tackling obesities: Future choice – Project report. (2007). United Kingdom Government Office for Science: London, United Kingdom. (Online): https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/287937/07-1184x-tackling-obesities-future-choices-report.pdf (PDF 10.3MB).
Biography:
 |
Toby Lowe PhD (https://www.linkedin.com/in/toby-lowe/) is a visiting professor of Public Management, Centre for Public Impact Europe, on secondment from Newcastle Business School, Northumbria University, Newcastle, UK. He is interested in public management/administration, complexity and systems thinking. |
Article source: How measuring impact gets in the way of real world change. Republished by permission.
Header image source: Created by Bruce Boyes with Perchance AI Photo Generator.
Editor’s note: See also Simplistic solutions to complex problems turns behavioural science into a dangerous pseudoscience.
