Critical thinking
Article: Kievit, Frankenhuis, Waldorp, & Borsboom (2013)
Simpson's paradox in psychological science: a practical guide

Introduction

Simpson’s paradox: the direction of an association at the population-level may be reversed within the subgroups comprising that population.

Simpson showed that a statistical relation observed in a population could be reversed within all of the subgroups that make up that population.

What is Simpson’s paradox?

Simpson’s paradox is a counter-intuitive feature of aggregated data, which may arise when (causal) inferences are drawn across different explanatory levels. (like population to subgroup or subgroup to individual).

Simpson’s paradox is conceptually and analytically related to many statistical challenges and techniques.
The underlying shared theme of these techniques is that they are concerned with the nature of (causal) inference. The challenge is what inferences are warranted based on the data we observe.

Simpson’s paradox in individual differences

One can only be sure that a group-level finding generalizes to individuals when the data are ergodic, which is a very strict requirement.
Since this requirement is unlikely to hold in many data sets, extreme caution is warranted in generalizing across levels.
The dimensions that appear in a covariance structure analysis describe patterns of variation between people, not variation within individuals over time.

A person X may have a position on five dimensions compared to other people in a given population, but this does not imply that person varies along this number of dimensions over time.

Two variables may correlate positively across a population of individuals, but negatively within each individual over time.

A survival guide to Simpson’s paradox

Simpson’s paradox may occur in a wide variety of research designs, methods, and questions.
There is no single mathematical property that all instances of SP have in common. Therefore, there will not be a single, correct rule for analysing data so as to prevent cases of SP.

What we can do is consider the instances of SP we are most likely to encounter, and investigate them for characteristic warning signals.

The most general danger of psychology is that we might incorrectly infer that a finding at the level of the group generalizes to subgroups, or to individuals over time.

Preventing Simpson’s paradox

Develop and test mechanistic explanations

The first step in addressing SP is to carefully consider when it may arise.

The mechanistic inference we propose to explain the data may be incorrect.
This danger arises when we use data at one explanatory level to infer a cause at a different explanatory level.

The first step is explicitly proposing a mechanism, determining at which level it is presumed to operate, and then carefully assessing whether the explanatory level at which the data were collected aligns with the explanatory level of the proposed mechanism.

Study change

One of the most neglected areas of psychology is the analysis of individual changes through time.
Most psychological research uses snapshot measurements of groups of individuals, not repeated measures over time.

Time-series data allows for the study of aggregate patterns.

Intervene

If we want to be sue the relationship between two variables at the group level reflects a causal pattern within individuals over time, the most informative strategy is to experimentally intervene within individuals

To answer the causal question, we need to devise an experimental study.
To model the effect of some manipulation, and therefore rule out SP at the level of the individual, the strongest approach is a study that can assess the effects of an intervention, preferably within individual subjects.