Using multiple lines and levels of evidence (MLLE) to generate knowledge for causality inference
Being able to infer cause and effect is a critical aspect of evidence-based policy1. Is there adequate evidence to be able to conclude that a particular intervention causes a particular outcome?
As part of the Australian Productivity Commission 2009 roundtable on the topic of Strengthening Evidence-Based Policy in the Australian Federation2, Professor Patricia Rogers presented on the topic of evidence effectiveness3. She advises that, as shown in Figure 1, approaches to evidence-based policy need to be situational, especially in terms of whether an intervention has complicated or complex aspects. Evidence availability can be improved through different knowledge processes.
For complex interventions, which need to be constantly adaptive, Rogers argues that the classic linear approach to evidence-based policy is inappropriate. Rather,
For interventions with important complex aspects, evidence-based policy involves developing broader principles to guide this adaptive practice, and for continuously learning and disseminating evidence, while recognising its limited generalisability and rapid obsolescence.
This different view of the evidence-based policy process can be represented by the model shown in Figure 2.
Rogers warns that while evidence from randomized controlled trials (RCTs) can make a valuable contribution to policy, RCT data can provide poor quality evidence of effectiveness. A corollary to this is that non-RCT data can provide good quality evidence of effectiveness. These other types of evidence can be rigorously analysed using either the general elimination methodology or the multiple lines and levels of evidence (MLLE) approach:
MLLE involves bringing together different types of evidence, and systematically analysing the strength of the causal argument linking an intervention or a cause and its effects. This analysis might consider the strength of the observed relationship, specificity, temporality, coherence with other accepted evidence, plausibility, analogy with similar interventions, biological plausibility,dose and consistency of association. Given the specialist and often cross-disciplinary nature of the scientific evidence, the investigation is undertaken by a panel of credible experts, spanning a range of relevant disciplines, who are asked to judge the credibility of the evidence and the causal analysis.
A “line of evidence” is a type of evidence that is investigated in relation to an intervention, and a “level of evidence” is a strength-of-evidence value used to determine the case for inferring causation.
The outcomes of the MLLE assessment can be fed into the action cycle shown in Figure 2, leading to the continuous improvement of interventions.
Implementing the MLLE approach
Comprehensive resources are available to assist the implementation of the multiple lines and levels of evidence (MLLE) approach:
- Causal Criteria Methods Manual – Methods for applying the multiple lines and levels of evidence (MLLE) approach for addressing questions of causality4
- Environment NSW Evaluation Design
- Environment NSW Recording assessment using MLLE criteria template.
MLLE has been successfully used in a number of human and ecological assessments. The human assessments include the Final Evaluation of the Improved Global Governance for Hunger Reduction Programme5 (see page 6) and Review of UN SWAP Evaluation Performance Indicator Reporting6 (see page 7). The ecological assessments include Determining and managing environmental flows for the Shoalhaven River7 (see page 50) and Review of the 1999 ACT Environmental Flow Guidelines8 (see Appendix D).
References:
- Zussman, D. (2003). Evidence-based policy making: Some observations of recent Canadian experience. Social Policy Journal of New Zealand, 64-71. ↩
- Productivity Commission (2010). Strengthening Evidence Based Policy in the Australian Federation, Volume 1: Proceedings. Roundtable Proceedings, Productivity Commission, Canberra. ↩
- Rogers, P. J. (2010). Learning from the evidence about evidence-based policy. In Strengthening Evidence Based Policy in the Australian Federation, Volume 1: Proceedings. Roundtable Proceedings, Productivity Commission, Canberra. ↩
- Norris, R., Nichols, S., Ransom, G., Liston, P., Barlow, A., and Mugodo, J. (2008). Causal Criteria Methods Manual Methods for applying the multiple lines and levels of evidence (MLLE) approach for addressing questions of causality. eWater Cooperative Research Centre & Institute for Applied Ecology, School of Resource Environment & Heritage Science, University of Canberra. ↩
- Food and Agriculture Organization of the United Nations (2016). Final evaluation of the Improved Global Governance for Hunger Reduction Programme – Annexes. April 2016. ↩
- Barnes, J. and Bishop, J. (2016). Review of UN SWAP Evaluation Performance Indicator Reporting. Prepared ImpactReady LLP for Review Management Group, UNEG Working Group on Gender Equality and Human Rights. ↩
- Boyes, B. (2006). Determining and managing environmental flows for the Shoalhaven River, Report 1 – Environmental Flows Knowledge Review. NSW Department of Natural Resources, May 2006. ↩
- Ogden, R., Davies, P., Rennie, B., Mugodo, J. and Cottingham, P. (2004). Review of the 1999 ACT Environmental Flow Guidelines. A report by the CRCFE to Environment ACT. November 2004. ↩
Stuart French posted to facebook that I might be interested in this piece Bruce. And, yes, It is an interesting and thoughtful piece. You touch on matters which need much greater scrutiny and attention. Sadly, I don’t see much really thoughtful reflection on this sort of stuff in the way that you have described and touched up. I wil look up some of your links when I get a chance.
Only two reflection points as a contribution to discussion. I have always been slightly uncomfortable with the idea of “evidence based policy”. I myself prefer to use the term Evidence informed policy. And, I would also add that this subtle distinction is grounded in notions of different types of reasoning that contribute to knowledge creation – inductive, deductive and abductive reasoning.
I tried to write stuff about this a few years ago under a section called “Principles underpinning the harnessing of evolutionary possibility” in this paper on national regulatory systems (see figure 2 in particular).
Thanks for sharing
https://minerva-access.unimelb.edu.au/bitstream/handle/11343/28941/265202_265202.pdf?sequence=1&isAllowed=y
Many thanks Richard for your comment, and the link to the “Cities, human well-being and the environment” paper.
I’ve actually been working on the shared context problem solving approach put forward in the paper for a long time, and can attest to the success of the approach. I’ve written a number of RealKM Magazine feature articles discussing research related to this approach, and presenting case studies.
These articles are: