Critical thinking
Article: Marewski, & Olsson, (2009)
Beyond the null ritual, formal modeling of psychological processes

Beyond the null ritual

Rituals can be characterized by a range of attributes including:

• Repetitions of the same action
• Fixations on special features such as numbers
• Anxieties about punishments for rule violations
• Wishful thinking

Each of these characteristics is reflected in null hypothesis significance testing.

One good way to make theories more precise is to cast them as formal models.
In doing so, researchers can move beyond the problems of null hypothesis significance testing, and simple difference searching.

What is a model?

In the broadest sense, a model is a simplified representation of the world that aims to explain observed data.
A model is a formal instantiation of a theory that specifies the theory’s predictions. This category also includes statistical tools, such as structural equation or regression models.
Statistical tools are not typically meant to mirror the workings of psychological mechanisms.

What is the scope of Modeling?

Modeling is not meant to be applied equally to all research questions. Each method has its specific advantages and disadvantages.

Modeling helps researchers answer involved questions and understand complex phenomena.
In psychology, modeling is especially suited for basic and applied research about the cognitive system.

Advantages of formally specifying theories

Four closely interrelated benefits of increasing the precision of theories by casting them as models:

• Models allow the design of strong tests of theories
• They can also sharpen research questions
• Models can lead beyond theories built on the general linear model
• Modeling helps to address real-world problems

Designing strong tests of theories

Models provide the bridge between theories and empirical evidence.
They enable researchers to make competing quantitative predictions, which in turn lead to strong comparative tests of theories.
Any quantitative prediction can be systematically better or worse than any other.

But, as soon as one starts to compare quantitative predictions from different models, the use of null hypothesis testing can become inappropriate or meaningless.

Sharpening research questions

Null hypothesis tests are often used to evaluate verbal, informal theories.
But, in such theories are underspecified, then they can be used post hoc, to ‘explain’ almost any observed empirical pattern.
In the worst case, they become one-word explanations, labels that are broad in meaning and vague, and therefore, provide little or not specification of the underlying mechanisms or theoretical structure.

Formal quantitative predictions are often not easy to grasp by intuitive reasoning.
Frequently, the predictions being made by models can only be understood by running computer simulations.
Often it is only when one starts modeling that one learns what a theory really predicts, and what it cannot account for, pointing to overlooked research problems.

The goal of modeling is not only to find out which of competing explanations for data is preferred, but also to sharpen the questions to be asked.

Going beyond linear theories

Many null hypothesis significance tests are only suited for the evaluation of simple hypotheses.
This, in turn, encourages the formulation of hypotheses hat can be subjected to such tests.

Even though a general linear model represents a precise methodological tool, it might not be the best starting point for theory building.

Using more externally valid designs to study real-world questions

Just as the general linear model and null hypothesis significance tests are often inadequate for conceptualizing and evaluating a theory, factorial designs can lead to tests of theories under conditions that have little to do with the world outside of the laboratory, which is where the explanatory power of theories should be proved.

Factorial designs can destroy the natural covariation of variables, making it difficult to generalize from them to a world where organisms exploit the confounded relations between different pieces of information.
A lack of external validity may be one reason why much basic research in psychology is of little use in the applied, real world.
Modeling allows researchers to deal with natural confounds without destroying them, they can be built into the models.

Modeling provides ways to increase the precision of theories.
In doing so, it helps researchers to quantify the explanatory power of their ideas, allowing them to select between competing accounts of data without having to rely exclusively on the null ritual.

The problem of overfitting

To conclude that one model provides a better account of data than another based on R² or other standard goodness-of-fit indices might be reasonable if psychological measurements were noise free.
In science, noise-free data are practically impossible to obtain.
Researchers are confronted with the problem of disentangling the variation in data that is the result of noise from the variation that is the result of the psychological process of interest.
Goodness-of-fit measures alone cannot make this distinction. As a result, a model can end up overfitting the data, it can also capture the variance that comes from error.

Generalizability: the ability of a model to predict new data. The degree to which it is capable of predicting all potential samples generated by the same process, rather dan to fit only a particular sample of existing data.

The degree to which a model is susceptible to overfitting is related to the model’s complexity. That is, a model’s inherent flexibility that enables it to fit diverse patterns of data.
Two factors contribute to a model’s complexity:

• The number of free parameters it had
• How the parameters are combined in it (its functional form)

The dilemma is: increased complexity makes a model more likely to end up overfitting the data while its generalizability to new data decreases.
At the same time a model’s generalizability can also increase positively with the model’s complexity, but only to the point at which the model is complex enough to capture systematic variations in the data. Beyond that point, additional complexity can result in decreases in generalizability, because then the model may also start to absorb random variations in data.

A good fit to existing data does not necessarily imply good generalizability to new data, which can make it hard to tell which of two models provides a better explanation for data.

Other pitfalls of model selection

There are many complications that can arise when designing and testing models

• The irrelevant specification problem.
  If precision is the major virtue of modeling, it can also be a curse. Modelers need to decide how to bridge the gaps between informal verbal descriptions of theories and formal implementations, which can lead to unintended discrepancies between theories and their various formal counterparts.
• Bononi paradox
  When models become more complex and realistic they become less understandable and more opaque.
• The identification problem
  For any behaviour there may exist a universe of different models all of which are equally capable of reproducing and explaining behaviour. As a result, it appears unreasonable to ask which of many models or theories is more ‘truthful’. Rather, one needs to ask which model is better than another given a set of criteria.

Sometimes, different criteria may not unanimously favour the same model, making it difficult to determine which model is the best.
Model selection is no ritual with fixed guidelines. It is an exercise that requires careful decision making on the part of the researcher.