Taking responsibility for complexity (section 2.3): Where problems can be tackled: distributed capacities and intelligence
This article is section 2.3 of a series of articles featuring the ODI Working Paper Taking responsibility for complexity: How implementation can achieve results in the face of complex problems.
Complex problems and the challenges they pose. This section [Section 2] should enable the reader to assess whether their implementation challenge is in fact a ‘complex’ problem, and to identify key characteristics to mark out the appropriate tools for managing the type of complexity faced. It first describes what is meant by a complex problem, and then outlines three specific aspects of complex problems that cause problems for traditional policy implementation. It goes into detail on each of these aspects, providing explanations and ideas to help the reader identify whether their policy or programme is complex in this way (Sections 2.3-2.5).
2.3 Where problems can be tackled: distributed capacities and intelligence
Some issues are relatively similar at different scales or levels: decisions can be taken at the top or the aggregate level, and action can be coordinated by passing instructions through well-defined, smoothly functioning hierarchies. In some contexts and in the face of some implementation challenges, however, problems manifest themselves in different ways at different levels, and decision-makers at one level have only a limited understanding of the dynamics of a problem for which they have responsibility. Moreover, rather than one organisation or ministry being fully in control of meeting a particular objective, this may require the collaboration of a variety of actors, which are truly ‘controllable’ by the ministry to differing extents. Action begins to resemble an ongoing negotiation, or a ‘negotiated order.’ 1
For example, while a policy-maker in a national ministry may have the mandate to address internal migration, she/he may be unable to fully understand the multiple effects of the issue on different areas as well as regional policy-makers do, or to understand the factors that shape migrants’ decisions as well those of people from their local community. Migration is also influenced by companies providing employment in or outside the country and sometimes by diaspora communities. Decisions must nonetheless be made and action taken to implement policies and programmes, even where it is difficult for centrally formulated policies to respect the realities of the diverse contexts in which they are to be applied, and where it is not possible for the ministry to be fully in control of what goes on.
This is about where knowledge resides, and where and in what ways different levels should be linked in to decision-making action. Complexity science highlights how there is often untapped potential for change in lower-level decision-making units, and for groups of actors not linked by formal hierarchies to ‘self-organise’ and work coherently towards a common goal. Lower levels can sometimes address complex problems more effectively for three main reasons. First, they often have a strong understanding of the local dynamics, with long-built and often largely tacit knowledge of the drivers of behaviour and how issues relate to these. This may be more pronounced in developing country contexts, where state agencies frequently do not have the resources, reach or context-dependent skills to adapt to the variety of different localities they must deal with2. Lower levels may lack important capacities and knowledge, but in the face of this type of complexity policy implementation that gives them space for some autonomy is likely to be able to capitalise on their insights.
Second, when rights and responsibilities in addressing a problem are given to lower levels, it is more likely they will have ownership over solutions. Where there is limited knowledge of lower levels, decision-makers looking to shape implementation are faced with a chain of principal–agent relationships, as even formalised analyses are unlikely to capture fully the tacit understandings of actors at different levels. With a variety of alternate principals and agents at each level, it will be extremely difficult to ensure effective implementation from the top without the genuine commitment of lower levels. At these levels, management dialogues can function in a more constructive way (as opposed to being trapped by political expediencies and grand rhetoric). Individuals and organisations based in a local area are more likely to have a higher stake in providing sustainable solutions to local problems, and are more likely to be able to build up the kinds of decision-making space in which tacit knowledge of particular locations can be drawn on and integrated with other perspectives.
Collaboration and collective action, built on natural patterns of social capital and founded on trust, are central to achieving sustainable change in the face of these kinds of problems. An agency should not impose a course of action by itself, but instead should work with and influence others. This has a ‘horizontal’ and a ‘vertical’ component.’ Vertically, action occurs at a number of different scales, with interactions between multiple levels of governance that must be taken into account. For example, resilience theory suggests that ignoring the effects of one scale of action on another is the most common reason for policy failure3; in addition, the literature on managing common pool resources (CPR) suggests that establishing rules at one level, without rules at other levels, will provide an incomplete system4. There is also a horizontal component, in that power and responsibilities are often distributed and overlapping between various actors at the same scale.
Boundedly rational actors face complex and uncertain situations, for example ones where they are without solid knowledge of the structure of a common resource and how different actions are likely to affect it; they are without clear signals about the likely behaviour of others whose behaviour is interdependent with theirs; and the climate and weather are also inherent unpredictable. A large number of development problems arise because people cannot take the necessary collective action5. In these situations, there needs to be the building of trust, on the tails of which can come incrementally better institutions for collaboration6. Trust is built more easily through face-to-face interactions and contact7,8, and proximity is usually necessary (but not sufficient) for constructive interaction9. Genuine participatory processes are central to ensure social capital between networks of actors affected by a problem. Devolution of power can promote reciprocity and cooperation, reinforce perceptions of common problems and common interests and strengthen shared values and identities.
There have been a number of calls to enable action at lower levels. For example, experience with natural resource management (NRM) shows that such approaches are most effective at lower levels. Studies into where NRM arrangements are most resilient and effective show it is important to match the scale of governance with that of the ecosystem10, and UN Environment Programme (UNEP) principles state that management should be decentralised to the lowest appropriate level. The importance of cooperation has been reemphasised: strong levels of trust are possibly the most important factor promoting beneficial cooperation and economic success11. It may be that people acting on the belief that there are benefits to cooperation (rather than seeing life as a ‘zero sum game’) is a major force for development.
Even in the face of collective action problems, that is, where it might be that uncoordinated actors at lower levels pursue individual interest ahead of longer-term payoffs for all, it is clear that traditional policy analysis has previously underestimated the potential for self-organisation12. Ostrom13 and others’ work on CPR shows that surprisingly large numbers of individuals and organisations facing such problems do self-organise, forming cooperative solutions at the small to medium scale so as to manage their resources for collective, sustainable benefits14. Insights from this field of work are likely to be relevant for a variety of CPR problems, as well as for organising collective action to provide public goods at some scales15.
These dynamics are witnessed in a wide variety of situations, and models of administration and management are beginning to learn how to harness them. For example, in the private sector, a number of companies have embedded mechanisms for ‘organisational participation’ – the direct involvement of employees in management decision-making16. Sometimes implemented as part of ‘high commitment’ policies, this has been found to help protect comparative advantage in the context of rapid change and in markets characterised by a requirement for high-quality, specialised products17. A review of the evidence on this shows that participation can enhance efficiency and effectiveness by resulting in better decisions (employees often have information management does not); because people are more likely to implement decisions they helped make themselves; by enhancing motivation and commitment; by improving communication and cooperation; and by improving relationships among different levels of a hierarchy.
Unsuitability of traditional tools
Many traditional approaches are not well-suited to these kinds of situation. Unfortunately, irrespective of the most appropriate level or scale to ensure an effective response to a problem, the bulk of the effort of policy-making is often carried out at a high level within central government departments; otherwise, an intervention is designed and negotiated within international organisations. Inputs from lower levels and scales are not heavily considered at the design stage, and implementation is seen as a matter of providing instruction and direction to lower levels in the chain of command – which will be steered to meet top-level goals. An organisation’s planning and implementation often focuses on those parts of a problem which can be strictly controlled by them, and evaluation concentrates on what can be attributed to their actions alone. For example, national governments often hold power for the management of particular ecosystems, such as conservation areas in forests, rather than allowing for meaningful contributions from actors at a district or municipality level.
Rather than negotiating action at multiple interlinked levels in response to problems that manifest themselves simultaneously in a number of different ways at a number of different scales, actors at lower levels are treated purely as a means to an end. An overriding focus on ‘upwards accountability’ means that performance is judged according to external, objective standards and rewards or sanctions are distributed accordingly. Popular accountability and management tools cast the relationships between donors and partners in a bureaucratic and contractual light, based on plans, budgets and accounts18. Problems experienced in the development sector in trying to integrate participatory tools into common practice illustrate the issues here (see Box 3). A recent policy paper by the Australian government makes the following argument:
Traditional hierarchical modes of decision-making, sequential approaches to problem solving and single points of hierarchy reflect the techniques and values of the industrial era in which they were developed; but governments are facing new policy challenges, difficult to identify and solve, multiple causes interacting in complex and poorly understood ways19.
Box 3: Lessons from experiences with participation
Decades of experience in attempting to incorporate participatory tools into the work of development agencies shows how they have fit poorly with dominant modes of implementation and have been marginalised and sidelined by traditional tools. They have overwhelmingly tended to be employed in an ‘instrumental’ manner, as tools to help achieve one’s own objectives20. Rather than fundamentally altering long-established ways of working, participatory development has largely become a set of techniques and technical measures to be used to help meet existing goals.
This has functioned as another input into scientific management processes, fitting neatly into the project cycle: the scope and objectives of a project are defined, then participatory approaches involve an engagement between the agency and local people to help decide how implementation can best meet these.
Worse, participation is sometimes ‘derisory’: trust and compliance are ascertained simply to build a perceived institutional legitimacy, which in turn it is hoped will engineer compliance with agency-driven decisions, objectives and goals21. In the long term, these practices serve to undermine the trust that is the basis for the functioning of even instrumental participation. Social capital and functioning collaborative institutions cannot be formed where one actor is using a participatory process as a way of furthering her/his own goals and is aiming to build trust just for this reason, regardless of the knowledge or priorities of others.
Unfortunately, in the context of complex problems, implementing policies based on systems of compliance and control not only is undesirable but also tends to work only superficially, with the reality of the situation concealed from view. On the face of it, we might see compliance with the contractual stipulations of log frame reports (etc.). In fact, contracted organisations are filling in such reports just to please the donors, with little pretence of representing reality, while they get on with what they see as the ‘real work’ quite separately from the pile of reviews they must deliver in order to receive their funding22. Organisational performance frameworks tend to impose unwanted reporting requirements on partners, become paper exercises to fulfil corporate requirements rather than honest assessments of progress and lessons learnt and put pressure on individual organisations to attribute results to themselves (rather than collective efforts)23.
Similarly, the issue of trust and local ownership does not disappear with externally imposed solutions, but may just become less visible. Lipsky24 shows how the implementation phase of policy-making is the most crucial determinant of success in developing countries, and that it is at this stage that actual policy is shaped most decisively, with a great deal of power in the hands of ‘street-level bureaucrats.’ In addition, without some kind of cooperation from local communities, the costs of ensuring compliance with policy will be significantly higher. It is important that there be trust in government staff and policies as fair and effective, so that policies can be workable.
By failing to draw on the knowledge of, or to achieve buy-in from, lower levels, policies that seem to provide a workable solution to a problem may turn out not to respect the reality of the varied contexts in which they are supposed to be applied, and can frequently be simply irrelevant to the ways the issue is experienced. Decisions may not be taken up at lower levels, and/or may be circumvented in various unforeseen ways. Moreover, opportunities to lead more effective action are not visible within these approaches to implementation – there tends to be an underlying assumption that lower levels and the multitude of actors within the remit of the policy-making body cannot overcome collective action problems to find their own solutions to issues25.
Even well-intentioned attempts to step in can make matters worse. Not only are opportunities to engage in the kind of collaborative action required to address these complex problems often lost, but also agencies using the ‘command and control’ framework can do serious harm to emergent forms of collaboration and action. For example, national interventions to improve agricultural production in Indonesia actually caused crop failures and famine until local organisation was recognised and integrated into action (see Box 4). Harm can be done by bypassing and ignoring existing (legitimate) organisations and networks, or by simply not recognising them as such. For example, the Africa Power and Politics Programme (APPP) suggests that dominant ‘blueprint’ approaches tend to see indigenous forms of governance as a problem to be replaced by the ‘ideal’ (Western) model; failure of these ‘good governance’ initiatives can be explained by the fact that they do not respect deep-rooted institutions, cultural norms and values that shape the societal response to development issues26.
Box 4: Hindering, and then helping, rice farming in Indonesia
In the 1970s, the Indonesian government decided that rice farming practices in Bali needed to be modernised as part of a massive redirection of agricultural policy. From their perspective, this was a story of the Green Revolution: acting on advice from the Asian Development Bank (ADB), they sought to introduce new, high-yielding varieties of rice, at the same time as legally mandating double- and triple-cropping of these new varieties. Despite some resistance, by 1977 70% of rice terraces in south central Bali were planted with the new varieties of rice.
Unfortunately, these policies failed to meet their most basic central aim: productivity and output dropped, with crop losses approaching 100%, as a result of explosions of pest infestations. In response to these, the government introduced further new crop varieties, but new pests then emerged. District government offices reported chaos in water scheduling and irrigation flows. Other side-effects were also seen (there has been a strong critique of the social and distributional impacts of the Green Revolution).
Meanwhile, the Balinese system of water temples, considered a minor obstacle by those government policymakers who had any knowledge of it at all, in fact held together complex, decentralised institutional arrangements for managing rice farming27. Managing agriculture in Bali is a complex issue with multiple interdependencies. Rice is grown in paddy fields fed by irrigation systems that depend on rainfall, which varies by season and elevation and, in combination with groundwater inflow, determines river flow.
By controlling the flow of the river into terraced fields, farmers are able to create pulses in important processes. For example, the cycle of wet and dry phases alters soil pH, with numerous effects on the quality of the soil. Cooperation between farmers with adjacent terraces can create a sufficiently large fallow area to deprive rice pests of their populations, but if too many farmers follow an identical process there may not be enough water for all, especially as there are weirs only every few kilometres. Water sharing and pest control are opposing constraints – an optimal level of coordination depends on local conditions. As such, parallel to the physical system of terraces and irrigation works is an equally complex social system, based on networks of temples dedicated to agricultural deities and the Goddess of the Lake. Farmers associations linked into these temples use the larger network to coordinate cropping patterns and phases of agricultural labour28.
Informed and inspired by a generation of anthropological fieldwork, Lansing and Miller29 use a simple game-theoretic model to simulate interactions between farmers and their environment, illuminating the emergence of cooperative behaviour among Balinese farmers30. The ‘fitness value’ or payoff of different farming strategies changes as a result of complex interactions between irrigation networks and the domesticated ecology of the rice terraces. A spontaneous process of self-organisation occurred when temples were allowed to react to changing environmental conditions over time in a simulation model. Cooperative networks emerged that bore very close resemblance to actual temple networks. As these networks formed, average harvest yields rose to a new plateau. Subsequently, the irrigation systems that were organised into networks were able to withstand ecological shocks such as pest outbreaks or drought much better than those that lacked networks. The networks have a definite structure, which leads to higher sustained productivity than would be the case if they were ordered randomly.
As such, water temple networks are a self-organised system for cooperation and collaboration between farmers, shaped by a process of agents co-evolving to a changing environment. Green Revolution policies, blind to these self-organised local arrangements, disrupted existing patterns of interaction, which explains their catastrophic failure in the Bali context. The destruction of the temple system by the ADB-sponsored government programme is now remembered by the Balinese farmers as a time of poso (hunger and harvest failure). The government now supports the role of water temples in pest control, but the episode is an important illustration of the dangers of applying the wrong approach in the wrong context. Approaching agricultural reform as primarily a technical matter missed the importance of structures for coordination and collaboration, and disregarding local knowledge and understandings meant policies did more harm than good.
Even where a ministry or agency does attempt to engage with local institutions, further harm can be done to emergent collaboration. ‘Programmed failure,’ whereby agencies force actors to work to unrealistically optimistic goals or timeframes, can play a strong role in degrading trust and genuine involvement31. In other cases, participation and consultation are undertaken simply in order to engineer compliance. A vast amount of research shows that human behaviour is based on norms of ‘strong reciprocity’ 32; where people see others as violating cooperative norms, the best that can be hoped for is them sticking around to make sure they ‘get their due’ from the process. More likely is their non-engagement.
Another insight comes from a study on capacity, change and performance33, which highlights that the way capacity develops is not amenable to more technocratic and rational approaches, in which organisations are viewed as machines and capacity in terms of ‘gaps,’ which are often put down to resource shortfalls. ODI’s work on facilitating networks has produced similar findings about the dangers of ‘blueprint’ or ‘deficit-based’ approaches, which fail to recognise and engage with existing networks of interactions (whether formal or informal), instead trying to create or control structures that cannot be created or controlled by one actor34.
Next part (section 2.4): When to take key decisions: unpredictability and emergent change processes.
See also these related series:
- Exploring the science of complexity
- Planning and strategy development in the face of complexity
- Managing in the face of complexity.
Article source: Jones, H. (2011). Taking responsibility for complexity: How implementation can achieve results in the face of complex problems. Overseas Development Institute (ODI) Working Paper 330. London: ODI. (https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/6485.pdf). Republished under CC BY-NC-ND 4.0 in accordance with the Terms and conditions of the ODI website.
References and notes:
- Theories of ‘negotiated order’ say there is an interconnected, co-evolutionary relationship between formal and informal institutions, between shared beliefs and actualised behaviour and between micro-level behaviour and macro-level dynamics. This brings ideas from Marx and Bourdieu, among others, together into four tenets: 1) all social order is negotiated order – organisation is not possible without some form of negotiation; 2) specific negotiations are contingent on the structural conditions of organisation, i.e. they follow lines of communication and are not random; 3) negotiations have temporal limits, and they are renewed, revised and reconstituted over time; and 4) the structure of organisation and the micro-politics of the negotiated order are closely connected. In other words, it is important to look for the possible balances and interplays between the forces attempting to organise a system from the top down and the reactions of the agents within a system to each other and to the environment in which they act (see Callaghan, G. (2008). ‘Evaluation and Negotiated Order: Developing the Application of Complexity Theory.’ Evaluation 14(4): 339-411.). ↩
- Swanson, D. and Bhadwal, S. (eds) (2009). Creating Adaptive Policies: A Guide for Policy Making in an Uncertain World. Winnipeg and Ottawa: IISD and IDRC. ↩
- Lebel L. (2006). ‘Inclusive and Adaptive Governance.’ Working Paper WP-2010-01. Chiang Mai: USER. ↩
- Ostrom, E. (1999). Coping with tragedies of the commons. Annual review of political science, 2(1), 493-535. ↩
- Ostrom, E. (2005). Understanding Institutional Diversity. Princeton, NJ: Princeton University Press. ↩
- Ostrom, E. (1990). Governing the Commons: The Evolution of Institutions for Collective Action. Cambridge: Cambridge University Press. ↩
- Beinhocker, E. (2006). The Origin of Wealth: Evolution, Complexity, and the Radical Remaking of Economics. Cambridge, MA: Harvard Business Press. ↩
- Uphoff, N. (1992). ‘Local Institutions and Participation for Sustainable Development.’ Gatekeeper Series 31. London: IIED. ↩
- Axelrod, R. and Cohen, M. (2000). Harnessing Complexity: Organizational Implications of a Scientific Frontier. New York: Basic Books. ↩
- Berkes, F., Colding, J. and Folke, C. (2003). Navigating social-ecological systems: Building resilience for complexity and change. Cambridge, UK: Cambridge University Press. ↩
- Harrison, L. and Huntingdon, S. (2000). Culture Matters: How Values Shape Human Progress. New York: Basic Books. ↩
- Ostrom, E. (1990). Governing the Commons: The Evolution of Institutions for Collective Action. Cambridge: Cambridge University Press. ↩
- Ostrom, E. (1990). Governing the Commons: The Evolution of Institutions for Collective Action. Cambridge: Cambridge University Press. ↩
- Poteete, A., Janssen, M., and Ostrom, E. (eds) (2010). Working Together: Collective Action, the Commons, and Multiple Methods in Practice. Princeton, NJ: Princeton University Press. ↩
- Ostrom, E. (1990). Governing the Commons: The Evolution of Institutions for Collective Action. Cambridge: Cambridge University Press. ↩
- Heller, F., Pusic, E., Strauss, G. and Wilpert, B. (1998). Organizational Participation: Myth and Reality. New York: Oxford University Press. ↩
- Heller, F., Pusic, E., Strauss, G. and Wilpert, B. (1998). Organizational Participation: Myth and Reality. New York: Oxford University Press. ↩
- Roche, C. (1999). Impact Assessment for Development Agencies: Learning to Value Change. Oxford: Oxfam. ↩
- APSC (2009). ‘Delivering Performance and Accountability, Contemporary Government Challenges.’ Canberra: APSC. ↩
- Rudqvist, A. and Woodford-Berger, P. (1996). ‘Evaluation and Participation: some Lessons’, SIDA Studies in Evaluation 96/1. Stockholm: SIDA ↩
- Arnstein, S. (1969). ‘A Ladder of Citizen Participation.’ Journal of the American Institute of Planners 35(4): 216-224. ↩
- Bakewell, O. and Garbutt, A. (2004). The Use and Abuse of the Logical Framework Approach. Stockholm: Sida. ↩
- Smutylo, T. (2001). ‘Crouching Impact, Hidden Attribution: Overcoming Threats to Learning in Development Programs.’ Ottawa: Evaluation Unit, IDRC. ↩
- Lipsky, M. (1983). Street-level Bureaucracy. New York: Russell Sage Foundation. ↩
- Ostrom, E. (1990). Governing the Commons: The Evolution of Institutions for Collective Action. Cambridge: Cambridge University Press. ↩
- Kelsall, T. (2008). ‘Going with the Grain in African Development?’ Africa Power and Politics Programme Discussion Paper 1. London: ODI. ↩
- Lansing, J. and Miller, J. (2003). ‘Cooperation in Balinese Rice Farming.’ Santa Fe, NM: Santa Fe Institute. ↩
- Lansing, J. and Miller, J. (2003). ‘Cooperation in Balinese Rice Farming.’ Santa Fe, NM: Santa Fe Institute. ↩
- Lansing, J. and Miller, J. (2003). ‘Cooperation in Balinese Rice Farming.’ Santa Fe, NM: Santa Fe Institute. ↩
- Simulation models are uniquely appropriate for addressing issues of adaptation and determinism in the development of complex social systems. Extending the use of simulations in biology, which might focus on the co-evolution of algae and Antarctic sea ice, this analysis moved from natural ecosystems that evolved through a process of ‘blind’ natural selection. ↩
- Ruitenbeek, J. and Cartier, C. (2001). ‘The Invisible Wand: Adaptive Co-management as an Emergent Strategy in Complex Bio-economic Systems.’ Occasional Paper 34. Jakarta: CIFOR. ↩
- One of the strongest (most triangulated) findings for the complexity sciences is that, nearly universally, people are ‘conditional co-operators’ and ‘altruistic punishers.’ This has been explained as ‘do unto others as you would have them do unto you (conditional cooperation); if others don’t do unto you, nail them, even at personal cost to yourself (altruistic punishment).’ Studies of game theory and ABM have shown strong reciprocity to be a highly ‘fit’ evolutionary strategy for adaptive agents, and evidence from controlled experiments, empirical studies and anthropological fieldwork has confirmed its universality from Middle America to the Peruvian rainforest. There is even evidence to suggest a generic and biochemical basis for the behaviour. ↩
- Baser, H. and Morgan, P. (2008) ‘Capacity, Change and Performance.’ Discussion Paper 59B. Maastricht: ECDPM. ↩
- Mendizabal, E. (2008). ‘Supporting networks: Ten principles.’ Opinion. 105. July 2008. London: ODI. ↩
