Does using non-naive participants influence effect size in experiments?

Researchers often assume that the participants in their studies are naive to the research materials. In the case of first year university students, which are often used as research subjects, it is assumed that their prior exposure to research is limited. In the case of internet samples, which are also widely used, it is assumed that the pool of participants is large enough to offset the prior exposure of any participants.

However, even though having a new intake of students every year means that there is a fast turnover, the students are likely to be used by a number of researchers with similar interests. With internet samples, people can remain in the participant pool for several years, and responses tend to be dominated by a small number of “professional” survey takers. People can also be exposed to research materials through belonging to several participant pools, or gain knowledge of the materials from university courses, members of other participant pools, or coverage by the media.

Previous research suggests that participants’ prior exposure to research materials typically influences effect sizes in experiments, but this had not been directly tested. A recent study¹ addresses this gap.

It was found that prior exposure to research materials can reduce effect size, with effect sizes decreasing by about 25% when replicated on the same sample. The effect of this on experimental power can be surprisingly large, with substantial increases in sample size needed to compensate.

The findings of the study show that researchers need to more carefully recruit their research participants. The study authors warn that “Self-reported participation is an imperfect measure of prior participation. It does not identify all prior participants, or even those who demonstrated a particularly large behavioral effect of prior participation.”

Given this, the authors suggest that researchers need to directly monitor prior participation rather than just asking participants if they have participated before. If this isn’t possible, then procedures and stimuli different to those known to the tested population should be used, or sample size should be increased to compensate.

Reference:

Also published on Medium.