Critical thinking in Quasi-Experimentation - summary of an article by Shadish (2008)

Critical thinking
Article: Shadish (2008)
Critical thinking in Quasi-Experimentation

All experiments are about discovering the effects of causes.
All experiments have in common the deliberate manipulation of an assumed cause, followed by observation of the effects that follow.

A Quasi-experiment: an experiment that does not use random assignment conditions.

Causation

What is a cause?

An inus condition: an insufficient cause by itself. It effectiveness required it to be embedded in a larger set of conditions.

Most causal relationships are not deterministic, but only increase the probability that an effect will occur.
This is the reason why a given causal relationship will only occur under some conditions but not universally.
To different degrees, all causal relationships are contextually dependent, so the generalization of experimental effects is always at issue.

Experimental causes are manipulable.
Experiments explore the effects of things that can be manipulated.
Experimental causes must be manipulable.

In quasi-experiments, the cause is whatever was manipulated, which may include many more things than the researcher realizes were manipulated.
In quasi-experiments, especially if the researcher is not the person manipulating the treatment, it is easy to make mistaken claims about what was manipulated, and the context in which it occurred.

What is an effect?

In an experiment, we observe what did happen when people receive a treatment.
The counterfactual is knowledge of what would have happened to those same people if they simultaneously had not received treatment.

An effect is the difference between what did happen and what would have happened.

We can never observe the counterfactual.
Experiments try to create reasonable approximations to this physically impossible counterfactual.

Two central tasks in experimental design are:

• Creating a high-quality but necessarily imperfect source of counterfactual inference
• Understanding how this source differs form the treatment condition.

Random assignment forms a control group that is often the best approximation to this counterfactual that we can usually obtain, though even that control group is imperfect because the person in the control group are not identical to those in the treatment group.
However, we do know that participants in the treatment and control group differ form each other only randomly.

The problem in quasi-experiments is that differences between treatment and control are usually systematic, not random, so nonrandom controls may not tell us much about what would have happened to the treatment group if they had not received treatment.
Much of quasi-experimentation is concerned with creating good sources of counterfactual inference. In general, quasi-experiments use two different tools to do so

• Observing the same unit over time
• To try to make nonrandom control groups as similar as possible to the participants in the treatment group.

Even then, the effects of quasi-experiments are rarely as trustworthy as those form randomized experiments.

Causal relationship

A causal relationship exists if:

• The cause preceded the effect
• The cause was related to the effect
• We can find no plausible alternative explanation for the effect other than the cause.

Quasi-experiments improve over correlational studies in two ways:

• Quasi-experiments force cause to precede effect by first manipulating the presumed cause and then observing an outcome afterwards
• Quasi-experiments allow the researcher to control some (but not all) of the third variable explanations.

Campbell’s threats to valid causal inference

Campbell codified some of the most commonly encountered group differences into a more general system of threats to valid causal inference. Reasons why researchers might make a mistake when saying that a treatment caused an outcome.
For each of these threats, the subsequent list provides a label, the definition.
The reader should decide whether each of these causes are plausible.

• History
  Events occurring concurrently with treatment that could cause worse performance.
• Maturation
  Naturally occurring changes over time that could be confused with a treatment effect.
• Selection
  Systematic differences over conditions in respondent characteristics that could also cause the observed effect.
• Attrition
  A loss of respondents to treatment or to measurement can produce artefactual effects if that loss is systematically correlated with conditions.
• Instrumentation
  The nature of a measure may change over time or conditions in a way that could be confused with a treatment effect.
• Testing
  Exposure to a test can affect scores on subsequent exposures to that test, which could be confused with a treatment effect.
• Regression to the mean
  When units are selected for their extreme scores, they will usually have less extreme scores on other variables, which can be confused with a treatment effect.

It is neither feasible or desirable to rule out all possible alternative interpretations of causal relationship.
Instead, only plausible alternatives are of concern.
Part of the critical thinking inherent in quasi-experimentation is to identify these alternative explanations, sort out the plausible from the merely possible, and then demonstrate with logic and data whether or not these alternative explanations occurred and could explain the effect.

But, plausibility judgments are inherently fallible.
The focus on plausibility is a two-edged sword:

• It reduces the range of alternatives to the considered in quasi-experimental work
• Yet, it also leaves the resulting causal inference vulnerable to the discovery that an implausible-seeming alternative may later emerge as likely causal agent.

Campbell’s list is very general, but threats are often context specific, so that different substantive areas develop their own lore about the alternatives that are so important they need to be controlled, even their own methods for doing so.

Critical thinking in quasi-experiments means showing alternative explanations are unlikely

Critical thinking involved in quasi-experimentation involves a complex quasi-experiment with many tests, each having different biases, and a program of research that combines the results of many quasi-experiments done by different authors in different times and place with different biases.