Generating evidence using the Delphi method
By Dmitry Khodyakov. Originally published on the Integration and Implementation Insights blog.
What is Delphi? How has the Delphi method stood up over time? How can the best of Delphi be adapted to new circumstances and problems?
The Delphi method is a group-based process for eliciting and aggregating opinion on a topic with a goal of exploring the existence of consensus among a diverse group of handpicked experts. The Delphi method was developed at the RAND Corporation in the early 1950s to obtain a reliable expert consensus, which is often used as a substitute for empirical evidence when it does not exist.
The four key characteristics of the Delphi method are:
- anonymity,
- iterative data collection,
- participant feedback, and
- statistical determination of group response.
As a result, Delphi has become best practice for quantifying the results of group elicitation processes. The method is used by different disciplines to make forecasts, identify research priorities, explore likely impacts of different policy options, develop performance metrics, and create clinical guidelines, among many other topics.
A typical Delphi process involves asking a set of questions in a series of rounds. Panelists, chosen for their expertise and experience, answer anonymously. Question rounds are followed by feedback on the results of the previous round that show participants how their answers compare to the rest of the panel. Panelists can then revise their responses and provide explanations of their new responses in future rounds. The process continues until consensus is reached or a pre-determined number of rounds is completed.
In 2023, we celebrate 60 years since the first peer-reviewed journal article that mentioned Delphi by name was published. Although the Delphi method is widely used today, it had a slow start. My research team’s recent bibliographic review of the use of the Delphi method over time identified about 20,000 articles that either used or discussed the Delphi method. We found that almost half of all articles were published in the 2010s and an additional third in the first two and a half years of the 2020s. Nearly two-thirds of all published articles have appeared in medical journals, compared to 15% in science and technology or social science journals.
Expert consensus achieved through the Delphi process is considered to be more reliable and objective than the opinion of a single expert because it maximizes the benefits of engaging a group of knowledgeable individuals. By not meeting in person and keeping responses anonymous, the method reduces the likelihood of groupthink, as well as the impact of dominant personalities. Nonetheless, anonymity has its drawbacks. Delphi panelists often want to know something about other panelists and engage with them directly to be able to contextualize their responses and potentially resolve differences in perspective via a discussion. This is the reason why modified-Delphi was created.
The RAND/UCLA Appropriateness Method
The RAND/UCLA (University of California, Los Angeles) Appropriateness Method, a modified-Delphi, was created in the 1990s. It consists of two rounds of ratings and a discussion round in between. An experienced moderator is needed to facilitate such discussions, which typically focus on areas of disagreement among panelists. By adding a discussion round, which is usually conducted in-person, modified-Delphi approaches violate one of the key characteristics of the Delphi method – participant anonymity. They also limit the number of panelists to no more than 18 to allow for meaningful engagement.
ExpertLens®
In the late 2000s, my RAND colleagues and I developed ExpertLens, an online approach to conducting modified-Delphi panels, that preserves participant anonymity and engages large and diverse groups of participants at the same time.
ExpertLens has four goals:
- to expand the pool of “expert” participants;
- to allow non-collocated participants to share their ideas and interact with each other anonymously through online discussion boards;
- to seamlessly integrate participants’ votes (quantitative data) and comments (qualitative data); and
- to employ statistical analysis as a way to make decisions based on the input from diverse groups.
As a group decision support system, ExpertLens draws upon the Delphi method in that a group of geographically distributed participants is asked to answer a series of questions twice. It also shares some features of Nominal Group Technique in that it allows for discussion among participants (albeit via an online discussion board), which takes place between the two question rounds. ExpertLens has the following in common with both methods: it is iterative, asks participants to provide judgments independently, and provides them with feedback on the group results before asking for a new independent judgment.
ExpertLens panels are not open to anyone. In addition to those with acknowledged expertise on the topic under consideration, ExpertLens panels often include stakeholders with diverse kinds of knowledge who can provide interesting and valuable insights. They may be professionals with expertise in domains related to the topic under consideration; individuals, such as patients or caregivers, with a lived experience related to the study topic; or people, such as community activists, who are traditionally marginalized in policy decision-making processes.
The ExpertLens process typically consists of three to four distinct rounds:
- In an optional Round 0, participants brainstorm ideas or express their positions on a topic in an open-ended format.
- In Round 1, participants answer pre-determined close-ended questions developed based on Round 0 input or created by the study organizers (often based on literature review results) and explain their responses.
- In Round 2, participants are provided with a statistical summary of their group responses and their own answers to Round 1 questions. They discuss Round 1 results via moderated, anonymous, asynchronous online discussion boards.
- In Round 3, participants re-answer the questions from Round 1 based on the information from Round 2 (ie., the summary of the group responses).
ExpertLens has been used in numerous studies, including the development of a national suicide prevention research agenda and the identification of strategies for reducing domestic abuse in the US armed forces.
ExpertLens is a full-service system supported by RAND researchers who help with study design, programming the study protocol within ExpertLens, and data analysis.
Questions, discussion, and insights into your experiences with the Delphi method are welcome.
To find out more:
Dalal, S., Khodyakov, D., Srinivasan, R., Straus, S. and Adams, J. (2011). ExpertLens: A system for eliciting opinions from a large pool of non-collocated experts with diverse knowledge. Technological Forecasting and Social Change, 78, 8: 1426-1444. (Online) (DOI): https://doi.org/10.1016/j.techfore.2011.03.021
Khodyakov, D., Grant, S., Kroger, J., Gadwah-Meaden, C., Motala, A. and Larkin, J. (2023). Disciplinary trends in the use of the Delphi method: A bibliometric analysis. PLoS ONE. 18, 8: e0289009. (Online – open access): https://doi.org/10.1371/journal.pone.0289009
The first journal article on the Delphi method is: Dalkey, N. and Helmer, O. (1963). An experimental application of the DELPHI Method to the use of experts. Management Science, 9, 3: 458–467. (Online) (DOI): https://doi.org/10.1287/mnsc.9.3.458
More information about ExpertLens can be found at the following link (please note, the link will redirect to RAND): https://www.expertlens.org
Biography:
 |
Dmitry Khodyakov PhD is Senior Behavioral/Social Scientist at the RAND Corporation in Santa Monica, California, USA. At RAND, he is Director of ExpertLens, Co-Director of the Center for Qualitative and Mixed Methods, and Professor of Policy Analysis at Pardee RAND Graduate School (https://www.linkedin.com/in/dmitry-khodyakov/). His research focuses on methods of expert elicitation and stakeholder engagement, intervention/program evaluation, community-based participatory research, and ethics of stakeholder-engaged research. |
Article source: Generating evidence using the Delphi method. Republished by permission.
Header image source: Adapted from Clker-Free-Vector-Images on Pixabay.
