Systems and complexity

Using multiple lines and levels of evidence (MLLE) to generate knowledge for causality inference

Being able to infer cause and effect is a critical aspect of evidence-based policy1. Is there adequate evidence to be able to conclude that a particular intervention causes a particular outcome?

As part of the Australian Productivity Commission 2009 roundtable on the topic of Strengthening Evidence-Based Policy in the Australian Federation2, Professor Patricia Rogers presented on the topic of evidence effectiveness3. She advises that, as shown in Figure 1, approaches to evidence-based policy need to be situational, especially in terms of whether an intervention has complicated or complex aspects. Evidence availability can be improved through different knowledge processes.

Approaches to evidence-based policy
Figure 1. Approaches to evidence-based policy (source: Rogers 2010).

For complex interventions, which need to be constantly adaptive, Rogers argues that the classic linear approach to evidence-based policy is inappropriate. Rather,

For interventions with important complex aspects, evidence-based policy involves developing broader principles to guide this adaptive practice, and for continuously learning and disseminating evidence, while recognising its limited generalisability and rapid obsolescence.

This different view of the evidence-based policy process can be represented by the model shown in Figure 2.

Evidence uptake as an ongoing knowledge building process
Figure 2. Evidence uptake as an ongoing knowledge building process (source: Rogers 2010).

Rogers warns that while evidence from randomized controlled trials (RCTs) can make a valuable contribution to policy, RCT data can provide poor quality evidence of effectiveness. A corollary to this is that non-RCT data can provide good quality evidence of effectiveness. These other types of evidence can be rigorously analysed using either the general elimination methodology or the multiple lines and levels of evidence (MLLE) approach:

MLLE involves bringing together different types of evidence, and systematically analysing the strength of the causal argument linking an intervention or a cause and its effects. This analysis might consider the strength of the observed relationship, specificity, temporality, coherence with other accepted evidence, plausibility, analogy with similar interventions, biological plausibility,dose and consistency of association. Given the specialist and often cross-disciplinary nature of the scientific evidence, the investigation is undertaken by a panel of credible experts, spanning a range of relevant disciplines, who are asked to judge the credibility of the evidence and the causal analysis.

A “line of evidence” is a type of evidence that is investigated in relation to an intervention, and a “level of evidence” is a strength-of-evidence value used to determine the case for inferring causation.

The outcomes of the MLLE assessment can be fed into the action cycle shown in Figure 2, leading to the continuous improvement of interventions.

Implementing the MLLE approach

Comprehensive resources are available to assist the implementation of the multiple lines and levels of evidence (MLLE) approach:

MLLE has been successfully used in a number of human and ecological assessments. The human assessments include the Final Evaluation of the Improved Global Governance for Hunger Reduction Programme5 (see page 6) and Review of UN SWAP Evaluation Performance Indicator Reporting6 (see page 7). The ecological assessments include Determining and managing environmental flows for the Shoalhaven River7 (see page 50) and Review of the 1999 ACT Environmental Flow Guidelines8 (see Appendix D).

References:

  1. Zussman, D. (2003). Evidence-based policy making: Some observations of recent Canadian experience. Social Policy Journal of New Zealand, 64-71.
  2. Productivity Commission (2010). Strengthening Evidence Based Policy in the Australian Federation, Volume 1: Proceedings. Roundtable Proceedings, Productivity Commission, Canberra.
  3. Rogers, P. J. (2010). Learning from the evidence about evidence-based policy. In Strengthening Evidence Based Policy in the Australian Federation, Volume 1: Proceedings. Roundtable Proceedings, Productivity Commission, Canberra.
  4. Norris, R., Nichols, S., Ransom, G., Liston, P., Barlow, A., and Mugodo, J. (2008). Causal Criteria Methods Manual Methods for applying the multiple lines and levels of evidence (MLLE) approach for addressing questions of causality. eWater Cooperative Research Centre & Institute for Applied Ecology, School of Resource Environment & Heritage Science, University of Canberra.
  5. Food and Agriculture Organization of the United Nations (2016). Final evaluation of the Improved Global Governance for Hunger Reduction Programme – Annexes. April 2016.
  6. Barnes, J. and Bishop, J. (2016). Review of UN SWAP Evaluation Performance Indicator Reporting. Prepared ImpactReady LLP for Review Management Group, UNEG Working Group on Gender Equality and Human Rights.
  7. Boyes, B. (2006). Determining and managing environmental flows for the Shoalhaven River, Report 1 – Environmental Flows Knowledge Review. NSW Department of Natural Resources, May 2006.
  8. Ogden, R., Davies, P., Rennie, B., Mugodo, J. and Cottingham, P. (2004). Review of the 1999 ACT Environmental Flow Guidelines. A report by the CRCFE to Environment ACT. November 2004.

Bruce Boyes

Bruce Boyes (www.bruceboyes.info) is editor, lead writer, and a director of the award-winning RealKM Magazine (www.realkm.com) and currently also teaches in the University of NSW (UNSW) Foundation Studies program in China. He has expertise and experience in a wide range of areas including knowledge management (KM), environmental management, program and project management, writing and editing, stakeholder engagement, communications, and research. Bruce holds a Master of Environmental Management with Distinction and a Certificate of Technology (Electronics). With a demonstrated ability to identify and implement innovative solutions to social and ecological complexity, Bruce's many career highlights include establishing RealKM Magazine as an award-winning resource for knowledge managers, using agile and knowledge management approaches to oversee the implementation of an award-winning $77.4 million river recovery program in western Sydney on time and under budget, leading a knowledge strategy process for Australia's 56 natural resource management (NRM) regional organisations, pioneering collaborative learning and governance approaches to support communities to sustainably manage landscapes and catchments, and initiating and teaching two new knowledge management subjects at Shanxi University in China.

Related Articles

2 Comments

  1. Stuart French posted to facebook that I might be interested in this piece Bruce. And, yes, It is an interesting and thoughtful piece. You touch on matters which need much greater scrutiny and attention. Sadly, I don’t see much really thoughtful reflection on this sort of stuff in the way that you have described and touched up. I wil look up some of your links when I get a chance.

    Only two reflection points as a contribution to discussion. I have always been slightly uncomfortable with the idea of “evidence based policy”. I myself prefer to use the term Evidence informed policy. And, I would also add that this subtle distinction is grounded in notions of different types of reasoning that contribute to knowledge creation – inductive, deductive and abductive reasoning.

    I tried to write stuff about this a few years ago under a section called “Principles underpinning the harnessing of evolutionary possibility” in this paper on national regulatory systems (see figure 2 in particular).

    Thanks for sharing

    https://minerva-access.unimelb.edu.au/bitstream/handle/11343/28941/265202_265202.pdf?sequence=1&isAllowed=y

    1. Many thanks Richard for your comment, and the link to the “Cities, human well-being and the environment” paper.

      I’ve actually been working on the shared context problem solving approach put forward in the paper for a long time, and can attest to the success of the approach. I’ve written a number of RealKM Magazine feature articles discussing research related to this approach, and presenting case studies.

      These articles are:

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Back to top button