Threat

What happens?

When?

Solution

Maturation threat

A change in behaviour occurs more or less spontaneously over time. People adapt to changed environments.

One-group, pre-test/post-test design

Using a comparison group

History threat

A specific event has occurred between the pre-test and the post-test that affects almost every participant systematically (e.g: a change of seasons).

One-group, pre-test/post-test design

Using a comparison group

Regression threat

If a group’s mean is unusually extreme at the pre-test, it is likely to be less extreme at the post-test, closer to the typical mean (e.g: depressed people have an extreme mean of sadness and this probably will be less extreme when it is tested again). Regression alone does not make an extreme group cross over the mean to the other extreme.

One-group, pre-test/post-test design

Using a comparison group

Attrition threat

A reduction in participant numbers that occurs when people drop out before the end. This is only a problem if attrition is systematic.

One-group, pre-test/post-test design

Not using the scores of participants that dropped out

Testing threat

There is a change in the participants as a result of taking a test more than once (e.g: participants might become better at a test if they practice). Participants change over time.

Pre-test/post-test design

Abandoning the pre-test or using a comparison group.

Instrumentation threat

A measuring instrument changes over time (e.g: observers change the way they observe behaviour over time or two different instruments are used in the pre-test and the post-test that are not equivalent).

Pre-test/post-test design

Abandoning the pre-test, calibrating the instruments and using different groups of participants.

Selection-history threat

An outside event or factor affects only those at one level of the independent variable.

Pre-test/post-test design

Using comparison groups and checking whether an effect affects both groups

Selection-attrition threat

Only one of the experimental groups experiences attrition.

Pre-test/post-test design

Using comparison groups and not using the scores of participants that dropped out.

Observer bias

The researcher’s expectations influence their interpretation of the results.

Any study

Using a double-blind design or a masked design

Demand characteristics

Participants guess that the study is supposed to be about and change their behaviour in the expected direction.

Any study

Using a double-blind design or a masked design

Placebo effects

People receive a treatment and believe it will work and thus it has an effect.

Any study

Using a double-blind placebo control study with a control group.

Cause null effect

What happens?

Solution

Weak manipulation

The independent variable is not operationalized well enough.

Use a manipulation check

Insensitive measures

There is not an operationalization of the dependent variable that is sensitive enough.

Use a manipulation check

Ceiling- and floor effects

All participants score either very high or very low.

Use a manipulation check

Design confounds acting in reverse

A design confound counteracts and this works against any true effect of the independent variable on the dependent variable.

A proper design preventing design confounds

Noise (error variance / unsystematic variance)

There is too much unsystematic variability within each group. The greater the overlap between the members of two groups, the smaller the effect size.

Keeping the within-group variability to a minimum

Measurement error

A human or instrument factor inflates or deflates a person’s true score on the dependent variable.

Using reliable and precise tools and measure more instances (e.g: measure more often or measure more people participants)

Individual differences

There are a lot of individual differences between participants.

Changing the design (e.g: within groups instead of an independent-groups design) or add more participants

Situation noise

There are external distractions.

Carefully controlling the environment of an experiment