Critical thinking
Article: Dennis & Kintsch
Evaluating Theories

Introduction

A theory is a concise statement about how we believe the world to be.
Theories organize observations of the world and allow researchers to make predictions about what will happen in the future under certain conditions.
Science is about the testing of theories, and the data we collect as scientists should either implicitly or explicitly bear on theory.

The characteristics that lead a theory to be successful from those that make it truly useful:

• Descriptive adequacy:
  Does the theory accord with the available data?
• Precision and interpretability:
  Is the theory described in a sufficiently precise fashion that other theorists can interpret it easily and unambiguously?
• Coherence and consistency:
  Are there logical flaws in the theory? Does each component of the theory seem to fit with the others in to a coherent whole? Is it consistent with theory in other domains?
• Prediction and falsifiability:
  Is the theory formulated in such a way that critical tests can be conducted that could reasonably lead to the rejection of the theory?
• Postdiction and explanation:
  Does the theory provide a genuine explanation of existing results?
• Parsimony:
  Is the theory as simple as possible?
• Originality:
  Is the theory new or is it essentially a restatement of an existing theory?
• Breadth:
  Does the theory apply to a broad range of phenomena or is it restricted to a limited domain?
• Usability:
  Does the theory have applied implications?
• Rationality:
  Does the theory make claims about the architecture of mind that seem reasonable in the light of the environmental contingencies that have shaped or evolutionary theory?

Criteria on which to evaluate theories

Descriptive adequacy

The extent to which it accords with data.
In psychology, the most popular way of comparing a theory against data is null hypothesis significance testing.
Determining whether a theory is consistent with data is not always as straightforward as it may at first appear.

Some of the the subtleties involved in determining the extent to which a theory accords with data

• Using null hypothesis significance testing, it is not possible to conclude that there is no difference. A proponent of a theory that predicts a list-length effect can always propose that a failure to find the difference was a consequence of lack of power of the experimental design.
• Null hypothesis significance testing encourages a game of 20 questions with nature. A study proceeds by setting up one or more dichotomies. Rather than develop a cohesive theory and use hypothesis testing to evaluate it, researchers will generate an endless set of issues, each of which is only loosely coupled with previous work, and little cumulative progress will be achieved.

Precision and interpretability

When evaluating a theory, pay close attention to how well the constructs in the theory and the relationships between them are defined. Can you be confident that you could apply the theory in a related domain unambiguously?
Conversely, when articulating a theory, ask yourself what implicit assumption you may be making that may not be shared by your readers.

Coherence and consistency

Although it may seem obvious that theories should be free form logical flaws, it isn’t easy to spot these flaws, particularly when theories are presented in verbal form.
Another common problem that one must look for in evaluating theories is circularity.

Beyond ensuring that theories are free form logical flaws, it is important to ask how consistent a theory is, both with other theories within psychology and also with theory outside psychology.
Our current understanding remains our best guess of how the world operates, and so theories that are consistent with this approximation are more likely to endure.

Prediction and falsifiability

Ideally, one should be able to make unambiguous predictions based on the theory and conduct empirical tests that could potentially bring the theory into doubt.

Although falsification provides the most useful information in advancing scientific knowledge, it is sometimes the case that verifying predictions can increase our confidence in a theory.
However, not all predictions are equally diagnostic.

• Predictions that seem to violate our intuitions and yet turn out to be the case provide more support for a theory than predictions that are unsurprising.

Postdiction and explanation

Prediction is possible under well-controlled laboratory conditions, but hardly ever under natural conditions.
Postdictive explanations are weaker than predictive explanations, but they are still explanations.
Prediction cannot be our goal.
Explanation after the fact (postdiction) is possible and worthwhile.

To predict we need to understand what is going on and have sufficient control over the relevant variables.
Postdiction also implies understanding, though not control.
The true goal of science is understanding, not necessarily prediction.

Postdiction can be based on formal theories as much as prediction.

Parsimony (Occam’s Razor)

The principle of parsimony states that theories should be as simple as possible.
In choosing between models of a given phenomenon, we would like to favor the one that fits the data well and is as simple as possible.

Breadth

Theories should attempt to be as broad as possible, while maintaining the other criteria such as descriptive adequacy and the ability to provide genuine explanations of phenomena.

Originality

Great care must be taken when comparing theories against each other, even when they are stated formally.

Usability

Good scientific theories should be useful in addressing societal problems.

Rationality

Does the theory make claims about the architecture of mind that seem reasonable in the light of the environmental contingencies that have shaped our evolutionary history?