Fearing the future of empirical psychology: Bem's (2011) evidence of psi as a case study of deficiencies in modal research practice - LeBel & Peters - 2011 - Article

Introduction

Bem, a researcher, reported a series of nine experiments in the Journal of Personality and Social Psychology. He claimed these experiments to be proof for the existence of psi, which he defined as the anomalous retroactive influence of future events on an individual's current behavior. Lebel, the author of our article, uses Bem's article as a case study to discuss important problems with modal research practice (MRP). The MRP is the most accepted methodology that empirical psychologists most commonly use in their research. Lebel also discusses how to improve this methodology.

Lebel states that Bem's report are of high quality. Therefore, it is a good example to discuss the problems with MRP. He discusses the ways in which Bem's results are not due to the real effect of psi, but due to the deficiencies of MRP. Lebel focus on three methodological issues in Bem's article, which reflect a general deficiency in MRP: an overemphasis on conceptual replication; insufficient attention to verifying the integrity of measurement instruments and experimental procedures and; problems with the way that the null hypothesis significance is tested (NSHT). These deficiencies lead to an interpretation bias of the data in empirical psychology. According to Lebel, Bem was able to publish his article in a high quality journal, only because of the use of MRP instead of the real results.

Lebel states that he uses Bem's article as an example, but that his criticism and recommendations for improved practices, apply to all the research that is conducted within the MRP tradition.

The interpretation bias

Lebel states that empirical data undermine the choice of theory. This means that there are always alternative explanations of data. This is the case when the data supports the researcher's hypothesis and when the data does not support this hypothesis. However, in MRP, deficiencies lead to biases in the interpretation of data, which is called interpretation bias. Interpretation bias means that there is a tendency towards interpretation of the data in a, for a researcher, favorable way. this is true whether the hypothesis is supported or not. Regardless of the data, the theory that is set up in MRP is not falsifiable. Therefore, there is an increased risk of reporting false positives and disregarding true negatives. Eventually, this leads to wrong conclusions about human psychology.

Lebel states that this interpretation bias has nothing to do with bad intentions of the researcher. It is merely the methodology that is deficient. He states that it is a systematic bias of MRP.

Conservatism in theory choice

Lebel discusses the problem of 'theory choice' in science to explain how the MRP has serious deficiencies. He explains that the knowledge system in sciences such as psychology are divided in two types of beliefs: theory-relevant beliefs (which are about the theoretical mechanisms that produce behavior), and method-relevant beliefs (which are about the procedures through which data are produced, measured and analyzed). In any experiment in which is a hypothesis, the interpretation of data relies on both types of beliefs. So, the data can be interpreted as theory relevant or as method relevant.

Lebel states that deficiencies in MRP systematically bias: the interpretation of confirmatory data as theory relevant; and the interpretation of disconfirmatory data as method relevant. This means that the researcher's hypothesis is protected from being falsificated. Lebel states that this is not good: the interpretation of data should not be dependent on the beliefs that are about theory or method, but rather on how central these beliefs are. Central beliefs are beliefs on which many other beliefs and theories are built. Peripheral belief are beliefs which not many beliefs are built on.

Deficiencies in MRP

Overemphasis on conceptual replication

Bem's nine experiments are 'conceptual replications'. This means that none of these experiments were replicated exactly. This is in line with MRP, in which a statistical significant result is followed by a conceptual replication with the goal of extending that underlying theory. However, when this conceptual replication fails, it is not clear whether this is due to the theory being false or due to methodological flaws during the replication. The most researchers choose for the latter when they produce a false conceptual replication. Then, they proceed with another conceptual replication, until they find a satisfactory significant finding.

So, the researcher has too much freedom in interpreting his or her results during conceptual replications. Also, the studies which count as replications are made after that they have been performed, and these choices are influenced by the interpretation bias. In other words, a successful replication is published, while many failed replications end up in the 'file drawer'.

Integrity of measurement instruments and experimental procedures

In Bem's article, he did not report anything about the integrity of the measurement instruments and the experimental procedures. He did not provide any reliability estimates for these measures and procedures. So, it is not clear whether these measurements are fit to measure the dependent variable in his article. This lack in verification of the integrity of the measurement instruments and experimental procedures weakens method-relevant beliefs. Lebel states that is indeed difficult to determine whether a manipulation (treatment) or measurement is a good one. This problem reflects the problem of construct validity in psychology. This problem arises because psychological processes are very context specific. This makes it difficult to know if a manipulation or measurement is indeed valid.

Problems with NHST

Bem uses the null hypothesis significance as the only criterion to determine his conclusion. Lebel state that this is not a good operation, because: the standard null hypothesis that there are no differences is almost always false, and it can lead to bizarre conclusions about the data. NHST, null hypothesis significance testing, is a general practice in MRP. In MRP, the null hypothesis is often formulated as a "nil hypothesis", which states that the means of different populations are identical. This is not a good hypothesis, because it is almost always false: differences between populations are inevitable. Lebel states that the use of NHST undermines the rigor of empirical psychology, because it increases the ambiguity of theory choice.

Strategies for improving MRP

Lebel's recommends strategies to improve research practice in psychology. He states that the methodology must be made more rigorous, by strengthening method-relevant beliefs.

Recommendations for strengthening method-relevant beliefs

Stronger emphasis on close replication

Lebel states that MRP would improve if there would be a stronger emphasis on close replication, compared to conceptual replication. Close replication is a core element of science. It determines whether an observed effect is real or is only due to sampling error. Close replication would only be achieved through "parallel experiments" and should lead to increased confidence after each replication.

Verifying integrity of methodological procedures

To make beliefs about methods stronger and thereby improve the validity of MRP findings, it is critical to verify the validity of the measurement instruments and procedures. Lebel recommends that there should be pilot studies which are explicitly designed to fine tune manipulations or measurement instruments. It should also be standard practice to check the internal consistency of the scores of a measurement instruments and to confirm measurement invariance of instruments across conditions. Lebel also recommends to develop an empirically supported account of how context sensitivity of mental processes vary under different conditions, to deal with the construct validity problem in psychology.

Use stronger forms of NHST

According to Lebel, a strong form of NHST should be used. For example, as is the case in astronomy and physics, significance tests should be treated as just one criterion. He argues for Bayesian analytic techniques, which incorporates base rate information into hypothesis testing.

Recommendations for weakening theory-relevant beliefs

Finally, Lebel states that psychological hypothesis should be set up in a way that could make disconfirmation possible.