Critical Thinking in Quasi-Experimentation - Shadish - 2008 - Article

Introduction

With experiments we manipulate an assumed cause and then we observe which effects do follow. This is used for all modern scientific experiments, we try to discover the effects that a cause generates. It is also used in quasi-experiments, but in that context we have to think critically about causation. In a quasi-experiment there is no random assignment used to conditions, but they are carefully chosen.

An Example of a quasi-experiment

For example, in a quasi-experiment children were chosen for the control group. They tried to create a control group that is the same, as much as possible, as the treatment group.

Causation

In our daily lives we mostly intuitively recognize causal relationships. Nevertheless, for many years a precise definition to cause and effect hasn’t been made by philosophers. The definitions depend partly on each other.

What is a cause?

When we look at causes, such as lightning or a lighted match, we can see that none of them is sufficient to generate the effects. We also need multiple other conditions. For example, a lighted match alone is not enough to start a fire, we also need oxygen, combustible materials, it has to be dry, and so on. We can call this lighted match an ‘inus condition’. This means that it’s an insufficient but still needed part of an unnecessary but sufficient condition. We need many factors for an effect to occur, but most of the time we don’t know all of them or their relation.

Experimental causes

For experimental causes the critical feature is that they are manipulable, otherwise we cannot deliberately vary them to discover what happens. When we look at quasi-experiments, the cause is the thing that was manipulated. It’s possible that the researcher doesn’t realize all the thing that were manipulated and that there are many more.

What is an effect?

Something that is the reverse to a fact is a counterfactual. With experiments, we observe what did happen, but for the counterfactual we look at what would have happened. And we can never observe this counterfactual. We still try to create an approximation to this unobservable counterfactual in experiments, and then we have to understand how the given source differs from the initial condition. Often, the best approximation is random assignment that forms a control group. Even though that this control group is not perfect, because the persons in the control group are not identical to those in the treatment group.

Counterfactuals in quasi-experiments

In quasi-experiments the differences between treatment and control are normally not random, but systematic. So these nonrandom controls may not tell us much about what would have happened. These quasi-experiments make use of two different tools to do so. The first one is observing the same unit over time and the second tool is to try to make the nonrandom control groups as identical as possible to the treatment group. But we do not know all the variables, there are always unknown differences. So this creates the problem of being not as good an estimate of the counterfactual as are random controls. 

Causal relationship

According to John Stuart Mill, we have a causal relationship if the cause happened before the effect, the cause related is to the effect, and we cannot find a plausible alternative explanation for the effect. But in most studies it is impossible to know which of the two variables came first. Quasi-experiments have two ways to improve this. First, they force the cause to come before the effect by first manipulating the presumed cause and then observing an outcome afterward. Second, they allow the researcher to control some of the third-variable alternative explanations. Nevertheless, the researches almost never knows what those third variables are.

Campbell’s threats to valid causal inference

Campbell (1957) has provided a tool to identify differences between the control and treatment groups. He codified some of the most commonly encountered group differences which give reasons why researchers might make a mistake in causation. But this list is very general, even though that the threats are often context specific. Part of the critical thinking used in quasi-experimentation is to identify the alternative explanations, to see if they are plausible, and then show whether or not these alternative explanations occurred and could explain the effect.

Critical thinking in quasi-experiments means showing alternative explanations are unlikely

Falsification, created by Popper, is to deliberately try to falsify the conclusion that you wish to draw, rather than only seek information corroborating them. These conclusions are plausible until shown otherwise. Quasi-experimentation also follows this logic. Experimenters have to identify a causal claim, and then they have to generate and examine plausible alternative explanations. But there are a two problems. The first is that the causal claim is never completely clear and detailed. So mostly the claim is just changed slightly when being falsified. Second,our observations are never perfect. They always reflect our wishes, so they can never provide definitive results. So we neither confirm or disconfirm the causal claim.