Join with a free account for more service, or become a member for full access to exclusives and extra support of WorldSupporter >>

Critical thinking in Quasi-Experimentation - summary of an article by Shadish (2008)

Critical thinking
Article: Shadish (2008)
Critical thinking in Quasi-Experimentation

All experiments are about discovering the effects of causes.
All experiments have in common the deliberate manipulation of an assumed cause, followed by observation of the effects that follow.

A Quasi-experiment: an experiment that does not use random assignment conditions.

Causation

What is a cause?

An inus condition: an insufficient cause by itself. It effectiveness required it to be embedded in a larger set of conditions.

Most causal relationships are not deterministic, but only increase the probability that an effect will occur.
This is the reason why a given causal relationship will only occur under some conditions but not universally.
To different degrees, all causal relationships are contextually dependent, so the generalization of experimental effects is always at issue.

Experimental causes are manipulable.
Experiments explore the effects of things that can be manipulated.
Experimental causes must be manipulable.

In quasi-experiments, the cause is whatever was manipulated, which may include many more things than the researcher realizes were manipulated.
In quasi-experiments, especially if the researcher is not the person manipulating the treatment, it is easy to make mistaken claims about what was manipulated, and the context in which it occurred.

What is an effect?

In an experiment, we observe what did happen when people receive a treatment.
The counterfactual is knowledge of what would have happened to those same people if they simultaneously had not received treatment.

An effect is the difference between what did happen and what would have happened.

We can never observe the counterfactual.
Experiments try to create reasonable approximations to this physically impossible counterfactual.

Two central tasks in experimental design are:

  • Creating a high-quality but necessarily imperfect source of counterfactual inference
  • Understanding how this source differs form the treatment condition.

Random assignment forms a control group that is often the best approximation to this counterfactual that we can usually obtain, though even that control group is imperfect because the person in the control group are not identical to those in the treatment group.
However, we do know that participants in the treatment and control group differ form each other only randomly.

The problem in quasi-experiments is that differences between treatment and control are usually systematic, not random, so nonrandom controls may not tell us much about what would have happened to the treatment group if they had not received treatment.
Much of quasi-experimentation is concerned with creating good sources of counterfactual inference. In general, quasi-experiments use two different tools to do so

  • Observing the same unit over time
  • To try to make nonrandom control groups as similar as possible to the participants in the treatment group.

Even then, the effects of quasi-experiments are rarely as trustworthy as those form randomized experiments.

Causal relationship

A causal relationship exists if:

  • The cause preceded the effect
  • The cause was related to the effect
  • We can find no plausible alternative explanation for the effect other than the cause.

Quasi-experiments improve over correlational studies in two ways:

  • Quasi-experiments force cause to precede effect by first manipulating the presumed cause and then observing an outcome afterwards
  • Quasi-experiments allow the researcher to control some (but not all) of the third variable explanations.

Campbell’s threats to valid causal inference

Campbell codified some of the most commonly encountered group differences into a more general system of threats to valid causal inference. Reasons why researchers might make a mistake when saying that a treatment caused an outcome.
For each of these threats, the subsequent list provides a label, the definition.
The reader should decide whether each of these causes are plausible.

  • History
    Events occurring concurrently with treatment that could cause worse performance.
  • Maturation
    Naturally occurring changes over time that could be confused with a treatment effect.
  • Selection
    Systematic differences over conditions in respondent characteristics that could also cause the observed effect.
  • Attrition
    A loss of respondents to treatment or to measurement can produce artefactual effects if that loss is systematically correlated with conditions.
  • Instrumentation
    The nature of a measure may change over time or conditions in a way that could be confused with a treatment effect.
  • Testing
    Exposure to a test can affect scores on subsequent exposures to that test, which could be confused with a treatment effect.
  • Regression to the mean
    When units are selected for their extreme scores, they will usually have less extreme scores on other variables, which can be confused with a treatment effect.

It is neither feasible or desirable to rule out all possible alternative interpretations of causal relationship.
Instead, only plausible alternatives are of concern.
Part of the critical thinking inherent in quasi-experimentation is to identify these alternative explanations, sort out the plausible from the merely possible, and then demonstrate with logic and data whether or not these alternative explanations occurred and could explain the effect.

But, plausibility judgments are inherently fallible.
The focus on plausibility is a two-edged sword:

  • It reduces the range of alternatives to the considered in quasi-experimental work
  • Yet, it also leaves the resulting causal inference vulnerable to the discovery that an implausible-seeming alternative may later emerge as likely causal agent.

Campbell’s list is very general, but threats are often context specific, so that different substantive areas develop their own lore about the alternatives that are so important they need to be controlled, even their own methods for doing so.

Critical thinking in quasi-experiments means showing alternative explanations are unlikely

Critical thinking involved in quasi-experimentation involves a complex quasi-experiment with many tests, each having different biases, and a program of research that combines the results of many quasi-experiments done by different authors in different times and place with different biases.

Image

Access: 
Public

Image

This content is used in:

WSRt, critical thinking - a summary of all articles needed in the second block of second year psychology at the uva

Search a summary

Image

 

 

Contributions: posts

Help other WorldSupporters with additions, improvements and tips

Add new contribution

CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Image CAPTCHA
Enter the characters shown in the image.

Image

Spotlight: topics

Check the related and most recent topics and summaries:
Institutions, jobs and organizations:
Activity abroad, study field of working area:
Countries and regions:
This content is also used in .....

Image

Check how to use summaries on WorldSupporter.org

Online access to all summaries, study notes en practice exams

How and why use WorldSupporter.org for your summaries and study assistance?

  • For free use of many of the summaries and study aids provided or collected by your fellow students.
  • For free use of many of the lecture and study group notes, exam questions and practice questions.
  • For use of all exclusive summaries and study assistance for those who are member with JoHo WorldSupporter with online access
  • For compiling your own materials and contributions with relevant study help
  • For sharing and finding relevant and interesting summaries, documents, notes, blogs, tips, videos, discussions, activities, recipes, side jobs and more.

Using and finding summaries, notes and practice exams on JoHo WorldSupporter

There are several ways to navigate the large amount of summaries, study notes en practice exams on JoHo WorldSupporter.

  1. Use the summaries home pages for your study or field of study
  2. Use the check and search pages for summaries and study aids by field of study, subject or faculty
  3. Use and follow your (study) organization
    • by using your own student organization as a starting point, and continuing to follow it, easily discover which study materials are relevant to you
    • this option is only available through partner organizations
  4. Check or follow authors or other WorldSupporters
  5. Use the menu above each page to go to the main theme pages for summaries
    • Theme pages can be found for international studies as well as Dutch studies

Do you want to share your summaries with JoHo WorldSupporter and its visitors?

Quicklinks to fields of study for summaries and study assistance

Main summaries home pages:

Main study fields:

Main study fields NL:

Follow the author: SanneA
Work for WorldSupporter

Image

JoHo can really use your help!  Check out the various student jobs here that match your studies, improve your competencies, strengthen your CV and contribute to a more tolerant world

Working for JoHo as a student in Leyden

Parttime werken voor JoHo

Statistics
2380