Brain power

Does using non-naive participants influence effect size in experiments?

Researchers often assume that the participants in their studies are naive to the research materials. In the case of first year university students, which are often used as research subjects, it is assumed that their prior exposure to research is limited. In the case of internet samples, which are also widely used, it is assumed that the pool of participants is large enough to offset the prior exposure of any participants.

However, even though having a new intake of students every year means that there is a fast turnover, the students are likely to be used by a number of researchers with similar interests. With internet samples, people can remain in the participant pool for several years, and responses tend to be dominated by a small number of “professional” survey takers. People can also be exposed to research materials through belonging to several participant pools, or gain knowledge of the materials from university courses, members of other participant pools, or coverage by the media.

Previous research suggests that participants’ prior exposure to research materials typically influences effect sizes in experiments, but this had not been directly tested. A recent study1 addresses this gap.

It was found that prior exposure to research materials can reduce effect size, with effect sizes decreasing by about 25% when replicated on the same sample. The effect of this on experimental power can be surprisingly large, with substantial increases in sample size needed to compensate.

The findings of the study show that researchers need to more carefully recruit their research participants. The study authors warn that “Self-reported participation is an imperfect measure of prior participation. It does not identify all prior participants, or even those who demonstrated a particularly large behavioral effect of prior participation.”

Given this, the authors suggest that researchers need to directly monitor prior participation rather than just asking participants if they have participated before. If this isn’t possible, then procedures and stimuli different to those known to the tested population should be used, or sample size should be increased to compensate.

Reference:

  1. Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological science, 26(7), 1131-1139.

Also published on Medium.

Bruce Boyes

Bruce Boyes (www.bruceboyes.info) is editor, lead writer, and a director of the award-winning RealKM Magazine (www.realkm.com) and currently also teaches in the University of NSW (UNSW) Foundation Studies program in China. He has expertise and experience in a wide range of areas including knowledge management (KM), environmental management, program and project management, writing and editing, stakeholder engagement, communications, and research. Bruce holds a Master of Environmental Management with Distinction and a Certificate of Technology (Electronics). With a demonstrated ability to identify and implement innovative solutions to social and ecological complexity, Bruce's many career highlights include establishing RealKM Magazine as an award-winning resource for knowledge managers, using agile and knowledge management approaches to oversee the implementation of an award-winning $77.4 million river recovery program in western Sydney on time and under budget, leading a knowledge strategy process for Australia's 56 natural resource management (NRM) regional organisations, pioneering collaborative learning and governance approaches to support communities to sustainably manage landscapes and catchments, and initiating and teaching two new knowledge management subjects at Shanxi University in China.

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Back to top button