Brain power

Does using non-naive participants influence effect size in experiments?

Researchers often assume that the participants in their studies are naive to the research materials. In the case of first year university students, which are often used as research subjects, it is assumed that their prior exposure to research is limited. In the case of internet samples, which are also widely used, it is assumed that the pool of participants is large enough to offset the prior exposure of any participants.

However, even though having a new intake of students every year means that there is a fast turnover, the students are likely to be used by a number of researchers with similar interests. With internet samples, people can remain in the participant pool for several years, and responses tend to be dominated by a small number of “professional” survey takers. People can also be exposed to research materials through belonging to several participant pools, or gain knowledge of the materials from university courses, members of other participant pools, or coverage by the media.

Previous research suggests that participants’ prior exposure to research materials typically influences effect sizes in experiments, but this had not been directly tested. A recent study1 addresses this gap.

It was found that prior exposure to research materials can reduce effect size, with effect sizes decreasing by about 25% when replicated on the same sample. The effect of this on experimental power can be surprisingly large, with substantial increases in sample size needed to compensate.

The findings of the study show that researchers need to more carefully recruit their research participants. The study authors warn that “Self-reported participation is an imperfect measure of prior participation. It does not identify all prior participants, or even those who demonstrated a particularly large behavioral effect of prior participation.”

Given this, the authors suggest that researchers need to directly monitor prior participation rather than just asking participants if they have participated before. If this isn’t possible, then procedures and stimuli different to those known to the tested population should be used, or sample size should be increased to compensate.

Reference:

  1. Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological science, 26(7), 1131-1139.
Rate this post

Also published on Medium.

Bruce Boyes

Bruce Boyes is a knowledge management (KM), environmental management, and education thought leader with more than 40 years of experience. As editor and lead writer of the award-winning RealKM Magazine, he has personally written more than 500 articles and published more than 2,000 articles overall, resulting in more than 2 million reader views. With a demonstrated ability to identify and implement innovative solutions to social and ecological complexity, Bruce has successfully completed more than 40 programs, projects, and initiatives including leading complex major programs. His many other career highlights include: leading the KM community KM and Sustainable Development Goals (SDGs) initiative, using agile approaches to oversee the on time and under budget implementation of an award-winning $77.4 million recovery program for one of Australia's most iconic river systems, leading a knowledge strategy process for Australia’s 56 natural resource management (NRM) regional organisations, pioneering collaborative learning and governance approaches to empower communities to sustainably manage landscapes and catchments in the face of complexity, being one of the first to join a new landmark aviation complexity initiative, initiating and teaching two new knowledge management subjects at Shanxi University in China, and writing numerous notable environmental strategies, reports, and other works. Bruce is currently a PhD candidate in the Knowledge, Technology and Innovation Group at Wageningen University and Research, and holds a Master of Environmental Management with Distinction and a Certificate of Technology (Electronics). As well as his work for RealKM Magazine, Bruce currently also teaches in the Beijing Foreign Studies University (BFSU) Certified High-school Pathway (CHP) program in Baotou, Inner Mongolia, China.

Related Articles

Back to top button