Brain power

Does using non-naive participants influence effect size in experiments?

Researchers often assume that the participants in their studies are naive to the research materials. In the case of first year university students, which are often used as research subjects, it is assumed that their prior exposure to research is limited. In the case of internet samples, which are also widely used, it is assumed that the pool of participants is large enough to offset the prior exposure of any participants.

However, even though having a new intake of students every year means that there is a fast turnover, the students are likely to be used by a number of researchers with similar interests. With internet samples, people can remain in the participant pool for several years, and responses tend to be dominated by a small number of “professional” survey takers. People can also be exposed to research materials through belonging to several participant pools, or gain knowledge of the materials from university courses, members of other participant pools, or coverage by the media.

Previous research suggests that participants’ prior exposure to research materials typically influences effect sizes in experiments, but this had not been directly tested. A recent study1 addresses this gap.

It was found that prior exposure to research materials can reduce effect size, with effect sizes decreasing by about 25% when replicated on the same sample. The effect of this on experimental power can be surprisingly large, with substantial increases in sample size needed to compensate.

The findings of the study show that researchers need to more carefully recruit their research participants. The study authors warn that “Self-reported participation is an imperfect measure of prior participation. It does not identify all prior participants, or even those who demonstrated a particularly large behavioral effect of prior participation.”

Given this, the authors suggest that researchers need to directly monitor prior participation rather than just asking participants if they have participated before. If this isn’t possible, then procedures and stimuli different to those known to the tested population should be used, or sample size should be increased to compensate.

Reference:

  1. Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological science, 26(7), 1131-1139.

Also published on Medium.

Bruce Boyes

Bruce Boyes (www.bruceboyes.info) is editor, lead writer, and a director of the award-winning RealKM Magazine (www.realkm.com), and a knowledge management (KM), environmental management, and project management consultant. He holds a Master of Environmental Management with Distinction, and his expertise and experience includes knowledge management (KM), environmental management, project management, stakeholder engagement, teaching and training, communications, research, and writing and editing. With a demonstrated ability to identify and implement innovative solutions to social and ecological complexity, Bruce's many career highlights include establishing RealKM Magazine as an award-winning resource, using agile and knowledge management approaches to oversee an award-winning $77.4 million western Sydney river recovery program, leading a knowledge strategy process for Australia's 56 natural resource management (NRM) regional organisations, pioneering collaborative learning and governance approaches to support the sustainable management of landscapes and catchments, and initiating and teaching two new knowledge management subjects at Shanxi University in China.

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Back to top button