Introduction

A false positive is likely the most costly error that can be made in science. A false positive is the incorrect rejection of a null hypothesis.

Despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not.

Many researchers often stop collecting data on the basis of interim data analysis. Many researchers seem to believe that this practice exerts no more than a trivial influence on the false-positive rates.

Solutions for authors

The authors of this article offer six requiremets for authors as a solution to the problem of false-positive publications:

1. Before the collection of data begins, authors must decide the rule for terminating data collection and they should report this rule in the article.
2. At least 20 observations per cell must be collected by the author or else the author should provide a compelling cost-of-data-collection justification.
3. All variables collected in a study must be listed.
4. All experimental conditions must be reported, including failed manipulations.
5. If observations are eliminate, authors must also report what the statistical results are if those observations are included.
6. Authors must report the statistical results of the analysis without the covariate, if an analysis includes a covariate.

Guidelines for reviewers

The authors of this article also offer four guidelines for reviewers:

1. Reviewers must make sure that authors follow the requirements.
2. Reviewers should be more tolerant of imperfections in results.
3. Reviewers must make possible that authors are able to demonstrate that their results do not hinge on arbitrary analytic decisions.
4. Reviewers should require the authors to conduct an exact replication, if justifications of data collection or analysis are not compelling.

Conclusion

The solution offered does not go far enough in the sense that it does not lead to the disclosure of all degrees of freedom. It cannot reveal those arising from reporting only experiments that ‘work’ (i.e., the file-drawer problem).

The solution offered goes too far in the sense that it might prevent researchers from conducting exploratory research. This does not have to be the case if researchers are required to report exploratory research as exploratory research. This also does not have to be the case if researchers are required to complement it with confirmatory research consisting of exact replications of the design and analysis that ‘worked’ in the exploratory phase.

The authors considered a number of alternative ways to address the problem of reasearcher degrees of freedom. The following are considered and rejected:

• Correcting the alpha levels. A researched could consider adjusting the critical alpha level as a function of the number of researcher degrees of freedom employed in each study.
• Using Bayesian statistics. This approach has many virtues, it actually increases researcher degrees of freedom by offering new set of analyses and by requiring to make additional judgments on a case-by-case basis.
• Conceptual replications. They are misleading as a solution to the problem at hand, because they do not bind researchers to make the same analytic decisions across studies.
• Posting materials and data. This would impose too high a cost on readers and reviewers to examine the credibility of a particular claim.

The goals of researchers is to discover the truth, and not to publish as many articles as they can. For different reasons researchers could lose sight of this goal.

The biblical story of Daniel encapsulates in a profound way the conduct of experimental science today. 'When King Nebudchadnezzar brought back thousands of captives, he wanted his followers to pick out the children who were not blemished and skillful in all wisdom. But there was a problem, because his favorite one, the boy named Daniel, refused to touch the food the King gave them out of religious reasons. The followers of the King were terrified because of this problem and how the King would react. But Daniel proposed a experiment; try for ten days giving him only vegetables and take another group of children and feed them the King's meat and wine. After ten days the two groups were compared. Daniel prospered the King's diet, and because of this and their healthy appearance, Daniel became the most important person of the Kingdom'. The followers make up a question about the causation, will a vegetarian diet cause servants to be healthy? Daniel proposes at his turn a methodology to deal with this question by comparing the two groups after ten days of experimenting. And after a suitable amount of time, you can see a difference between the two groups. Nowadays this is called a controlled experiment. 

You can't go back in time and see, when Daniel did eat the meat and wine, what will happen to him comparing to the healthy diet. But because you can compare Daniel with a group of people who will get a different treatment, you can see what will happen when you give people a different diet. But the groups need to be representative of the population and comparable with each other. 

But Daniel didn't think of one thing; confounding bias. Suppose that Daniel is healthier than the control group to start with, their robust appearance after the ten days of eating the healthy diet will have nothing to do with the diet itself. Confounding bias occurs when a variable influences both who is selected for the treatment and the outcome of the experiment. Sometimes the confounders are known as the 'lurking third variable'. 

But, statisticians both over- and underestimate the importance of adjusting for possible confounding variables. They overrate in the sense that they often control for many more variables than they need to and even for variables that they should not control for. The idea is 'the more things you control for, the stronger the your study seems', because it gives the feeling of specificity and precision. But sometimes, you can control for too much. 

Statisticians also underestimate the importance of controlling for possible confounding variables in the sense that they are loath to talk about causality at all, even if the controlling has been done correctly.

In this chapter you will get to know why you can safely use RCT's, randomized control trials, to estimate the causal effect X -> Y without falling prey to the confounder bias. 

What is meant with the 'chilling fear of confounding'? 

In 1998, an important study showed the association between regular walking and reduced death rates among retired men. The researcher wanted to know whether the men who exercised more lived longer. He found that the death rate over a twelve year period was two times higher among men who were a 'casual walker' (less than a mile a day) than among men who were 'intense walkers' (more than two miles a day). But you have to keep in mind the influence a confounding variable or bias might have. 

This classic causal diagram shows us that age is a confounder of walking and mortality. But also, maybe, physical condition could be a confounder. But by saying so, you can go on and on about what could be possible confounders. But, even if the researchers adjusted the death rate for age found that the difference between causal and intense walkers was still large.

The skillful interrogation of nature: Why do RCT's work?

An RCT is often considered the gold standard of a clinical trial, and the person to thank for this is R.A. Fisher. The question he asks are 'aimed at establishing causal relationships'. And what gets in the way is confounding. Nature is like a genie that answers exactly the question we pose, not necessarily the one we intend to ask. And around 1923/1924 Fisher began to realize that the only experimental design that the genie could not defeat was a random one. When you do an experiment multiple times, sometimes you may get lucky and apply it tot the most fertile subplots. But by generating a new random assignment each time you perform the experiment, you can guarantee that the great majority of the time you will be neither lucky nor unlucky. Now, the randomized trials are a golden standard, but in the time of Fisher an randomly designed experiment horrified him and all his statistical colleagues. But Fisher realized that an uncertain answer to the right question is much better than a highly certain answer to the wrong question. 

When you ask the genie the wrong question, you will never find out what you want to know. If you ask the right question, getting an answer that is occasionally wrong is much less of a problem. So, randomization brings two benefits:

1. It eliminates the confounder bias.
2. It enables the researcher to quantify his uncertainty. 

In a nonrandomized study, the experimenter must rely on her knowledge of the subject matter. If she is confident that her causal model accounts for sufficient numbers of deconfounders and she has gathered data on them, then she can estimate the effects in an unbiased way. But the danger here is that she might have missed a confounding factor, and her estimate may therefor be biased. 

But RCT's are still preferred to observational studies. In some cases, intervention may be physically impossible, or intervention could be unethical, or you have difficulties recruiting subjects for inconvenient experimental procedures and end up with only volunteers who don't quite represent the intended population.

What is the new paradigm of confounding?

While confounding is widely recognized as one of the central problems in research, a review of literature about this will reveal little consistency among the definitions of confounding or confounder. But why has the confounding problem not advanced a bit since Fisher? Because of lacking a principled understanding of confounding, scientist could not say anything meaningful in observational studies where physical control over treatments is infeasible. But how was confounding defined then and how is it defined now? It is easier to answer the second question, with the information we have now. Confounding can simply be defined as anything that leads to a discrepancy between the two P(Y|X) (the conditional probability of the outcome given the treatment) and P(Y |do(X)) (the interventional probability).But why is it so difficult? The difficulty is there because it isn't a statistical notion. It stands for the discrepancy between what we want to assess (the causal effect) and what we actually do assess using the statistical methods. But if you can't mathematically articulate what you want to assess, you can't expect to define what constitutes a discrepancy. 

The concept of 'confounding' has evolved around the two related conceptions: incomparability and lurking third variables. Both of these concepts have resisted formalization. Because how do we know what is interesting and relevant to study and to distinguish and what not? You can say it is common sense, but many scientist have struggled with finding the important things to consider. 

What are the two surrogate definitions of confounding? They fall into two main categories: declarative and procedural. A old procedural definition that goes by the scary name of 'noncollapsibilty'. You can compare the relative risk and the relative risk after adjusting for the potential confounder. And the difference indicates confounding and you should use the adjusted risk estimate. 

The declarative definition is 'the classic epidemiological definition of confounding' and it consists of three parts: A confounder of X (treatment) and Y (outcome) is a variable Z that is (1) associated with X in the population at large and (2) associated with Y among people who have not been exposed to the treatment X. In the recent years there has been supplemented a third condition: (3) Z should not be on the causal path between X and Y. But this idea is a bit confusing I would say. 

You can't always use Z as a perfect measure for M, when you do some of the influence of X on Y might 'leak through' if you control for Z. But controlling for Z is still a mistake, while the bias might be less if you controlled for M, it is still there. That is why Cox (1958) warned that you should only control for Z if you have a 'strong priori reason' to believe that it is not affected by X. This is nothing more than a causal assumption. 

Later, Robins and Greenland set out to express their conception of confounding in terms of potential outcomes. Also ideally, each person in their experiment would be exchangeable with the person in the other condition. So, you would have the same person in the treatment and in the control group that confounding variables could be very low. The outcome could be the same if you switched the treatments and controls. By using this idea, Robins and Greenland showed that both the declarative and procedural definition were wrong. 

What does the do-operator and the back-door criterion mean?

To understand the back-door criterion you first have to have an idea of how information flows in a causal diagram. It looks like links of pipes that convey information from a starting point X to a finish Y. The do-operator erases all the arrows that come into X and in this way prevents any information about X form flowing in noncausal direction. If you have longer pipes with more junctions:

A F -> G I -> J?

The answer is very simple, if a single junction is blocked, then J cannot 'find out' anything about A through this path. So, you have a lot of options to block communication between A and J. A back-door path is any path from X to Y that starts with an arrow pointing into X, X and Y will be deconfounded if we block every back-door path. So, you can almost treat deconfounding like some game. The goal of the game is to specify a set of variables that will deconfound X and Y. With other words: they should not be descended from X, and they should block all the back-door paths.

This is a new kind of bias, called the M-bias. There is only one back-door path, and this one is already blocked by a collider at B, so you don't need to control for anything else. It is incorrect to call a variable a confounder, like B, merely because it is associated with X and Y. B only becomes a confounder when you control for it! But when you are going to use identifying variables such as smoking, miscarriage etc. they are obviously not games but serious business that you are dealing with. 

Introduction

With so many different methods nowadays in psychology, it is not possible to be acquainted with all of psychology today. But when we look at the simpler times of psychology, we have two historic streams of method, which are widely used since the last century of our science: experimental psychology and correlational psychology. With both of these focussing on different things, the convergence of these two streams is still in the making.

The separation of the disciplines

Experimental psychology is when the scientist makes changes to the conditions to observe their consequences, and is the more coherent of the two disciplines. Correlational psychology also qualifies as a discipline because it asks a distinctive type of question and has technical methods to examine the question and the data. The correlator focuses more on the already existing variation between individuals, social groups, and species. Instead of the variation he himself creates, as the experimental psychologist does. With the experimental method it is a virtue that the variables are controlled, so it permits rigorous tests. While the correlator observes and organizes the data that nature has created. 

Characterization of the disciplines

Experimental psychology

In the beginning, experimental psychology was a replacement for naturalistic observation. With standardization of tasks and conditions they could get reproducible descriptions. Later the focus shifted to the single manipulated variable, and after that to multivariate manipulation. Another great development has been its concern with formal theory. But a problem that we do have in these experiments is due to individual variation. Because of this variation we get 'error variance'. We can reduce this error variance by selecting certain properties.

Correlational psychology

On the other hand, the correlational psychologist sees individual and group variation as important effects of biological and social causes. He wants to see what characteristics determine its mode and degree of adaptation, and his goal is to predict variation within a treatment.

The shape of a united discipline

It is not sufficient for each discipline to borrow from the other. A united discipline will study both experimental psychology and correlational psychology, but will also concern itself with the interactions between organismic and treatment variables. We should invent constructs and form a network laws which permits prediction. Investigators can use different methods, but still test the same theoretical propositions.

There already have been methodologies proposed for a joint discipline, one of the many choices is analytic procedures. Eventually, these two disciplines will become one, with a common theory, a common, method, and common recommendations for social betterment. A whole new dimension will be discovered and we will come to realize that organism and treatment are inseparable.

One of the most used rituals in science is that of the null hypothesis: testing the hypothesis versus chance of chance. Although it is known to be problematic, it is often used in practice. One way to resist the temptation of using the null hypothesis is to make the theories more precise by transforming them into formal models. These can be tested against each other instead of against chance, which in turn enables the researcher to decide between competing theories based on quantitative measures. 

The randomness of the .05 alpha level gives the writer flexibility in interpreting a p-value as an indication of proof against the null hypothesis. This article is about overcoming a ritual involved in testing hypotheses in psychology: the null ritual; or the null hypothesis significance testing.  

This means that a non-specific hypothesis is tested against "chance", or it says that "there is no difference between two population means." 40 years ago, editors of major psychological journals required this ritual to be carried out in order for the paper to be published. Although methodological evidence nowadays contradicts this, the .05 alpha level is still used.

What lies beyond the zero ritual?

Rituals have a number of attributes that all apply to the null hypothesis: A repetition of the same action, a focus on the 5% (or 1%) level, fear of sanctions from journal editors and wishful thinking about the results. In its most extreme form, the zero ritual reads as follows:

1. Set up a statistical null hypothesis with "no mean difference" or "zero correlation". Do not specify the predictions of the research hypothesis or of alternative hypotheses.

2. Use 5% as a convention for the rejecting the null. If it is significant, accept the research hypothesis.

3. Always perform this procedure.

Since this ritual became institutionalized in psychology, several alternatives have been proposed to replace or supplement it. The most of these suggestions focus on the way the data is analyzed. Think about effect size measures, confidence intervals, meta-analysis an resample methods. 

How is it possible that, despite attempts to introduce alternative ways, the null hypothesis is still being used, and the most? This may be due to the fact that most psychological theories are simply too weak to be able to do more than make predictions about directions of an effect. Therefore, we cannot offer an alternative form to the null hypothesis in this article, but a way to make theory more precise and to "make" it a formal model.

What is a model?

A model is a simplified representation of the world that is used to explain observed data: countless verbal and informal explanations of psychological phenomena. In a limited sense: a model is a formal instantiation of a theory that specifies the predictions.

What is the scope of modeling?

Modeling is not meant to always be applied in the same way. It must be seen as a tailor-made tool for specific problems. Modeling helps researchers to understand complex phenomena. Each method has its specific advantages and disadvantages of null hypothesis testing. Although it is also often used in other areas, it is often used in psychology for research on cognitive systems. Modeling is a complex undertaking that requires a lot of skills and knowledge.

What are the advantages of formally specifying theories?

There are four benefits of increasing precision of theories by casting them as models. 

1. A model provides a design that has strong theory tests

Models provide the bridge between theories and empirical evidence. They enable scientists to make competitive quantitative predictions that lead to strong comparative testing of theories. Making comparative testing of theories more precise ultimately leads to better systematic quantitative predictions between the theories tested. By comparing quantitative predictions with different models, the use of the null hypothesis testing can become unnecessary and useless.

2. A model can sharpen a research question

Null hypothesis are often used to test verbal, informal theories. But if such theories are not specified, they can be used post-hoc to "explain" every possible observed empirical pattern. Formal quantitative predictions are not easy to understand due to intuitive reasoning. The predictions that a model makes can only be understood by performing computer simulations. In summary, often it is only through self-modeling that someone understands what a theory actually predicts and what it cannot justify. The goal of modeling is not only to find out which of competing explanations for data is preferred, but also to sharpen the questions to be asked. 

3. A model can lead to the passing of theories that have arisen from the general linear model

Many null hypothesis significance tests only apply to simple hypothesis, for instance about linear addition effects. Scientists use available tools such as ANOVA and transform it into a psychological explanation for certain data. A prominent example is the attribution theory, which assumes that just as experimenters use ANOVA's to infer causal relations between two variables, outside the lab people infer causal relations by unconsciously doing the same calculation. But this might not be the best starting point for building a theory. Although general linear model (of which ANOVA is a way of) is a precise methodological tool, it is not always the best way to make statements or to base a theory on it.

4. A model helps to approach real-world problems

Just as the general linear model and a null hypothesis test are often inadequate when it comes to conceptualization and evaluation of a theory, factor designs can lead to testing theories under conditions that often have little to do with the real world, where the explanatory power of theories should be approved A lack of external validity can be one of the reasons why psychological outcomes make little contribution outside the lab in the real world: no person can randomly choose who they are in contact with, and no organism can "separate" correlations between life-sustaining information. But modeling, on the other hand, gives researchers the freedom to deal with natural confounders without destroying them: they can be built into the models. Modeling provides ways to increase the precision of theories. It helps researchers to quantify explanatory power. It ensures that they are not dependent on the null hypothesis. Formal statement can be linear and non-linear. By looking beyond factor designs, the possibility is created to approach real-world problems.

What are more benefits of formal modeling: an example of a modeling framework?

ACT-R is a wide, quantitative theory of human behavior that covers almost the entire human cognitive field.

Meta-analysis can be used to show that relying on significant tests slows the growth of cumulative knowledge. ACT-R, on the other hand, is a good example of how knowledge can systematically accumulate over time. ACT-R has its roots in old psychological theories, but in the end it knew its current form. This made it known that cognitive systems can give rise to adaptive processes by being transformed into static structures of the environment.

ACT-R models are specific enough for computer modeling for outcome and processes. For example, in a 2-alternative situation, reading this article or reading another article, an ACT-R model would predict which alternative would be chosen and what different reasons the model would consider before making this choice. Scientists can make the following predictions with ACT-R: (1) open behavior (2) temporal aspects of behavior (3) the associated patterns of an activity in the brain measured by fMRI scan.

In summary, modeling can promote the growth of cumulative knowledge, reveal how different behavioral activities are distributed and it can help to integrate psychological disciplines.

How do you select between competing formal models?

The comparison between alternative models is called model selection. There are a number of criteria for model selection: (1) psychological plausibility (2) falsifiability (3) number of assumptions that a model makes (4) whether a model is consistent with overarching theories (5) practical contribution. In practice, the criterion: descriptive adequacy is often used. This means that if 2 or more models are compared, the model that shows the least difference with existing data or the best fit is chosen.

A null hypothesis test is not a good way to choose between two models: if it has enough power, the test gives a significant result. But the biggest limitation of the model selection procedure based on significance or goodness-of-fit (R2) is that, on its own, these procedures do not approach the fundamental problem (choosing between two competing theories): overfitting.

What is the problem of overfitting?

To conclude that a model is better than the other using goodness-of-fit is reasonable if psychological measurements are noise-free. Unfortunately, noise-free data are practically impossible to obtain. As a result, there is an overfitting of the data: it not only captures the variance resulting from the cognitive process, but also that of a random error. Increased complexity causes a model to become overfit, thereby reducing generalizability. The generalizability is the degree to which the model is capable of predicting all potential samples generated by the same cognitive process, rather than to fit only a particular sample of existing data. The degree to which a model is susceptible to overfitting, is related to the model's complexity; the flexibility that enables it to fit certain patterns of data. 

At the same time, the generalizability of a model can positively increase the complexity of a model - but only to the point where the model is complex enough to include the systematic variations of the data. If that point is exceeded, it reduces generalizability because the model randomly absorbs variations in the data. A good fit does not necessarily ensure good generalizability for new data.

How do you select between models?

Practical

This approach relies on the intuition that when comparing models, one should choose the model that can best predict unobservable data. This can be done by testing the validity of a test (often the cross-validity test is done). A limitation of this test form is that it is not consistent. Another way to deal with this problem is to dose as many free parameters as possible. This can be done by fixing it or by creating simple models with few or no free parameters.

Simulation

By simulating the predictions of a competing model, one can gain insight into a specific behavior of a model. The results can be used to design the task to maximize the discriminatory power between models.

Theoretical

In this approach, the goodness-of-fit measurement is combined with theoretical estimates of model complexity that result in an estimate of generalizability. Generalizability (= goodness or fit + complexity) is based on the maximum log likelihood as a goodness or fit index. The complexity measurement takes different forms for different generalization measurements. The most commonly used approaches are: AIC and BIC. These are only sensitive to a kind of complexity: the number of parameters.

How do you choose between model selection approaches?

We cannot answer the question which is better, but we recommend to note the results of as many selection criteria as possible and to discuss the suitability of each criterion.

What are other pitfalls of model selection?

There are also other complications that can arise when you design and test models. If specification is the greatest virtue of modeling, it can also be the greatest curse. One must choose how a bridge can be made between informal verbal descriptions and formal implementations. This can lead to unintended discrepancies between theories and various formal counterparts. This is known s the irrelevant specification problem. 

A second problem that can arise in complex models is the Bononi paradox: when models become more complete and realistic they become less understandable and more opaque. 

Third, there can also be an identification problem. That for any behavior that exists there is a universe of different models that are all capable of explaining and reproducing the behavior. There are also an infinite number of vague and informal theories going around for which nobody will ever be able to decide whether one is better than another. 

Conclusion

Although modeling seems better from a scientific point of view, few use this approach to test. This is because it requires a lot of effort, time and knowledge. Accepting the null hypothesis test in laboratory settings leads to a reduction in the incentive for scientists to design models that explain issues in the "real" world. However, there is not often a better alternative to test and make models, because informal theories are not specific enough and they are tested against the chance of chance. With some knowledge and training, modeling can be performed with little effort.

What are the degrees of falsifiability?

A potential falsifier of a theory is any potential observation statement that would contradict the theory; for instance 'Peter the swan is black' is a falsifier of the hypothesis that 'all swans are white'. One theory can be more falsifiable than another if the class of potential falsifiers is larger. Therefore scientists prefer simple theories, because they are better testable. On the basis of not falsifiable theories Meehl criticized much psychology. 'Group A will score differently from Group B' also rules out virtually nothing and is a very weak theory. 

A theory can gain in falsifiability not only by being precise but also by being broad in the range of situations to which the theory applies. The greater the universality of a theory the more falsifiable it is, even if the predictions it makes are not very precise. 

Revisions to a theory may make it more falsifiable by specifying fine-grained causal mechanisms. As long as the steps in the proposed causal pathways are testable, specifying the pathway gives you more falsifiers.

Psychologists sometimes theorize and make predictions by constructing computational model, this is a computer simulation of a subject. In order for the model to perform, the free parameters have to be set in particular values, but you can't directly observe the values of these parameters. 

With computational models it can be difficult to predict how the model will do just by thinking about it. The model has to be run and its behaviour observed and analyzed. Often modellers just try to find any set of parameters values that fits the data. If the best-fitting model of each fitted about as wel, the modeler may conclude that there is no reason to prefer one model than another. Popper's idea showed that is inadequacy in simply finding the best-fit models. If a model has passed more severe tests it was more falsifiable to begin with. 

A theory that allows everything explains nothing, the more a theory forbids, the more it says about the world and the 'empirical content' of a theory increases with its degree of falsifiability. But also, the more falsifiable a theory is, the more it is open to criticism. So, the more falsifiable, the faster you can make progress, given progress comes from criticism. 

Popper stated that good science shows itself not just by the simple literal form of its theories, but also by the nature of history of its theories leading to the current proposals. When you have the hypothesis 'all swans are white', you can find one exception 'Peter the swan is black'. This amendment to the theory is adhoc and decreases the falsifiability and is unsatisfying. Popper proposed that revisions and amendments should always increase falsifiability. In psychology, attempts to save theories and to not make new measures are often called 'post hoc' .

Falsifiability: too strong a criterion or too weak?

There are two criticisms of Popper's approach:

1. No theory is falsifiable at all 
2. All theories are falsified anyway

Critics often focus on the fact that accepting an observation statement involves accepting various levels of theory as well as the theory under test. There is no general method of determining which of the theories should be rejected when an apparent falsification occurs. When you have a falsification, how do we know which component of the system to reject? This is a widely recognized problem and is called the Duhem-Quine problem. 

[toc]

Professor X sold 'football numbers'. The audience had to have a way of telling which football player was which, so each player had to wear a number on his football uniform. It didn't matter which number, as long as it wasn't more than a two-digit number.

Professor X loved numbers, when tests were made he couldn't wait to put the scores in his back pocket and hurry back to his office where he would lock the door, add the scores up, and then calculate the means and standard divisions for hours on end. Test scores are ordinal numbers, but ordinal numbers cannot be added. So, the professor came to the conclusion that it was wrong to compute means and standard deviations of test scores. This eventually lead to a nervous breakdown and Professor X retired. As an appreciation of his work, the school gave the Professor a 'football numbers' concession.

The Professor made a list of all the numbers given to him, and he found out he had 100,000,000,000,000,000 two-digit numbers to start with. The numbers were ordinal numbers, and the professor was tempted to add them up, square them and to compute means and standard deviations. But these numbers were only 'football numbers', such as letters of the alphabet. The teams brought there numbers to the professor, first the sophomore team, a week later the freshman team, but by the end of the week there was trouble. Information secretly reached the professor that the numbers in the machine had been tampered with. The freshman team appeared in person to complain. They said they had bought 1,600 numbers from the machine and complained that the numbers were too low and that they were laughed at because of the low numbers. 

The professor persuaded the freshman team to wait while he consulted the statistician who lived across the street. Maybe the freshman team got low numbers by chance. The statistician took the list of the professor, added them all together and divided them. 'The population mean', he said 'is 54,3'. But the professor expostulated 'You can't add them!'. 'Oh, I can't..? I surely just did it' said the statistician. 'But these are ordinal numbers, you can't add and divide them!'. 'The numbers don't know that' said the statistician. 'Since the numbers don't remember where they came from, they always behave just the same way, regardless'. And if you doubt my conclusions, I suggest you try and see how often you can get a sample of 1,600 numbers from your machine with a mean below 50,3 or above 58,3'. 

To date, the professor has drawn over 1,000,000,000 samples of 1,600 from his machine and only two samples were below 50,3 and above 58,3. He is happy because he is adding and dividing the football numbers that were given to him. 

Introduction

The implicit prejudice construct is a construct that has moved extremely fast in the recent years from psychology journals into other academic disciplines, newspaper editorials, courtrooms, boardrooms, and popular consciousness. This term appears for the first time in the PsycINFO database less than twenty years ago. A Google search for “implicit prejudice” between the years 1800 and 1990 would return only six hits, but in the years 1991-2006 it would return over 8400 hits. In Google Scholar, you would get 46 hits for the term “implicit prejudice” in articles published between 1800 and 1990. In the years after 1990, however, it would return 3700 hits. In 1998, an article about the implicit association test (IAT) was published. This is now the most popular method for studying implicit prejudice. The IAT article has been published more than 3700 times in PsycINFO, 3400 times in the Web of Science database and 7500 times in Google Scholar.

The authors of this article state that the psychological construct “implicit prejudice” is not a good construct. They state that even though it is a very popular construct, it is misunderstood and lacking in theory and in practical use. For example, scholars do not agree about how implicit prejudice is linked to other forms of prejudice. So, it is unclear if the different measures of the construct all measure the same construct. Also, the meaning of “implicit” is challenged by some researchers. When we look at the practical use of the term implicit prejudice, the results show that implicit measures of prejudice do not predict behavior better compared to traditional explicit measures of prejudice. In this article, they discuss how the psychological research behind the construct is inadequate to support the widespread use of the implicit prejudice. They explain by this by first discussing how the term was created and how psychologists marketed the ideas. Later, they discuss their view on how the construct should be renovated. They also discuss how such popular ideas within social psychology are hard to overcome.

The implicit prejudice

There are two eras in the history of the implicit prejudice construct. The first era is the pre-IAT era. In this era, psychologists mainly developed indirect measures of prejudice with the goal of overcoming response biases. They also mainly looked at automatic processes that lead to prejudice. The second era is the post-IAT era. This is the era in which the term became popular in public discussions and even in academic ones. In these discussions, people used the term implicit prejudice for other, widespread unconscious prejudices. Then, in 1998 an article was published in Psychology Today, which was stated that they found extraordinary results. By studying the unconscious, they found that everyone uses stereotypes, all the time without that people know it. In this article, they also stated that it is more difficult to avoid the negative effects of implicit prejudice compared to the negative effects of explicit prejudice. They author of the 1998 Psychology Today article states that: “Even though our internal censor successfully retains from expressing overly biased responses, there is still danger of leakage. This leakage shows itself in our non-verbal behavior, such as our expressions, our stance, how far we are from another person and how much eye contact we make.”

The IAT article introduced the idea that automatic stereotyping and unintentional prejudice operate in automatic and subconscious ways. The IAT is said to measure widespread implicit preferences that people may have for majority groups compared to minority groups. Even people who are from a minority group themselves, might show a preference for majority groups over minority groups. These preferences would be more predictive of behavior compared to explicit measures of prejudice.

The term “implicit prejudice” became very popular after 1998, because of the IAT article. The IAT Itself also became popular in the media and on the radio. For example, Greenwald discussed implicit bias and the behavioral effects of implicit bias on the radio. He said that: “People are not actually aware that they have this bias. But, it can still affect their behavior and act on them. It can produce unintended discrimination, because it can produce discomfort in interracial interactions.” The IAT was also discussed often in opinion websites, educational websites and in popular science writing.

The authors of this article, states that even though the IAT became very popular, it is not theoretically well grounded. Also, there have been no positive impacts of IAT research. The predictive validity of the implicit prejudice measure also has not shown to be higher than the predictive validity of explicit prejudice measures. The IAT also has not lead to any solutions for discrimination.

What Is implicit prejudice?

There are two important themes repeatedly found in tests the articles of the implicit prejudice. First, reaction-time based measures of prejudice, such as the lexical decision task (Wittenbrink, 1997) and the IAT, measure implicit prejudice, are meant to avoid the influence of normative pressures, such as the social desirability. The social desirability bias is the tendency of respondents to answer survey questions in a socially desirable way. Second, psychologists believe that they can tap into subconsciousness, decisions and behaviors. The implicit prejudice construct therefore also reflects views about attitudes and about the influence of automatic psychological processes and their influence on behavior. However, these themes are highly related to each other. This makes the authors of this article question: are the processes really implicit or is it only the measure that is an implicit measure of prejudice? This distinction between the meaning of “implicit” as in the processes or as in the measurement, is a highly discussed topic. The people who view the nature as the process, disagree about the nature of these processes. For example, IAT researchers state that implicit biases sometimes operate in subconscious ways. However, sometimes individuals are aware of their biases, but can simply not control them. The authors of this article state that it is also not clear if the two most reliable measures of prejudice, being the IAT and the “Affective Misattribution Procedure” (AMP), are implicit measures.

According to Greenwald & Nosek, the distinction between explicit and implicit prejudice is not empirically testable. Also, many automatic processes of prejudice and stereotyping, for example aversive racism, are included in the description of the implicit prejudice construct. However, the theory about aversive racism is different than the theory behind the implicit prejudice.

Another argument that the authors of this article give for their idea that the implicit prejudice construct need to be criticized, is that different implicit measures of implicit prejudice, produce different patterns of results. This is also true for similar measures. The measures also do not correlate highly.

The predictive validity of implicit prejudice measures

The IAT has face validity: this means that we think we know what it means when respondents say that they like one group more than another group. However, when the implicit prejudice is a measure of intergroup prejudice, it lacks face validity. The authors of this article state that for implicit measures of prejudice, predictive validity is crucial. Because, if what is measured by the IAT or another implicit measure reliably predicts behaviour that can lead to intergroup conflicts, then this would lead to better theoretical clarity about the underlying processes.

The measures of implicit prejudice are of an indirect nature. Many measures are reaction-time-based studies. In these studies, millisecond differences in response times, are seen as evidence for a prejudicial attitude. However, if these measures do not predict any judgments or behaviors, then this concept is meaningless.

The defenders of the implicit prejudice construct, posit as an argument for the construct that it predicts discriminatory behavior. One reviewer of Blindspot wrote when discussing the IAT: “The best indicator of the test’s validity is it’s prediction of behavior”. However, the authors of this article question whether the IAT predicts discriminative behavior better than explicit measures. This does not seem to be the case. Greenwald and colleagues (2009) reported in a meta-analysis that explicit measures predicted judgments, decisions and behavior better in seven of nine criterion domains. The IAT outperformed explicit measures in one area, but according to Greenwald and his colleagues this was due to poor performance of the explicit measures. The authors of this article also conducted an updated meta-analysis, based on that of Greenwald and colleagues. They found even lower estimates of the predictive validity for the IAT! The authors also found explicit measures to be more predictive than implicit measures in studies that looked at response times. This is in contrast with the idea that implicit construct resembles subtle, automatic behaviors compared to deliberate, controlled behaviors. The authors also found a lot of differences in the results across studies. They found that the variance was much greater than the estimated effect size.This means that the IAT will be a poor predictor of whether someone will act fairly or unfairly toward a minority group member. So, the popular ideas about IAT being better predictor of discrimination compared to explicit measures, is false. This is not true.

The score interpretation problem

According to IAT results, 75% of the American population is described as implicitly “racist”. According to the authors, there are no studies that provide criteria for the IAT scores to be nonexistent, low, moderate or high. So, the feedback given by the IAT is not based on the external validation. It is only based Cohen’s test of effect sizes. However, this test has the rule of thumb that effect sizes should be considered only on the base of practical meaning and significance. So, this is a noncorrect use of the Cohen’s effect size rule of thumb. According to the authors, if the IAT creators differently at their results, they could give the participants very different feedback. This has been shown earlier: the IAT researchers changed their criteria for the extremity of their scores. This lead to that the percentage of people showing strong anti-black bias on the IAT decreased from 48% to 27%. This change was not due to societal shifts or due to studies. It was solely due to the researchers’ change in definitions.

The authors state that because of this freedom of researchers to make important conclusions, it can be used in a negative way and lead to mischief. The conclusion of the authors is that the scores of the IAT have no clear meaning. This means that when one person scores higher on the IAT compared to another person, this does not mean that this person is more likely to express bias outside the testing context

The implicit sexism puzzle

Research has found that men usually do not exhibit sexism, while women do show pro-female implicit attitudes. This finding is not in line with the common findings in IAT. In the IAT, the findings are that the historically advantaged groups, men, are favored by both the members of this group and the disadvantaged groups. Therefore, the IAT is often used in explanations of female victimization. Because of this, new findings about sexism should be taken into account, but these findings have received little attention in the literature about the implicit prejudice construct and the IAT. The IAT researchers have responded to this finding by showing that men are more often associated with math and science and with limited leadership qualities. So, an IAT is often used to asses whether men or women are more often linked to science and math, but not to asses implicit attitudes toward men and women. According to the authors of this article, this focus on implicit gender stereotypes, is problematic because of practical and theoretical reasons. First, implicit measures of these stereotypes are not predictive of discriminatory behavior. Second, only a few implicit gender stereotypes have been examined. For example, research has not looked at whether traits of good managers such as cooperativeness, fairness and integrity are more strongly associated with women than men. So, the IAT does not measure all the stereotypes that there are. Also, there is no reason to believe that these implicit gender stereotypes are true for women for all positions.

Thirdly, there also seems to be a problem between implicit attitudes and automatic semantic associations. If prejudice and discrimination depend on how feelings and beliefs interact, then an interaction would be needed at the implicit level too. The authors also state that a contextualized model is needed, in which it is clear when and where each component of implicit prejudice will be predominant and in what behavioral form.

Bias is everywhere

According to Banaji and Greenwald, implicit attitudes are just evaluative associations of varying strengths with attitude objects. It does not matter if these objects are products, places or people and it also does not matter whether the source of these associations are cultural information or personal experiences. In this view, implicit prejudice is just an evaluative knowledge about different groups. Some evaluations are more negative than others.

The subjective judgment problem

One solution that is proposed to prevent the influence of implicit bias, is to objectify judgment and decision-making processes. This means that there should be only objective measures of performance. However, the articles positing this solution do not take into account the findings that these subjective evaluation criteria are not associated with discrimination against women and minorities.

The contributions of the IAT

The only positive of the IAT, according to the authors of this article, is that humans have implicit biases, at high prevalence rates. But, the bias categories of the IAT have no external validity. The IAT scores are only a reliable predictor of future IAT scores. Also, even though there has been a lot of research conducted into the implicit prejudice and the IAT, there is still a lot of confusion about the nature of implicit prejudice. In the practical use, claims and conclusions often depend on the interpreter of the results. The authors criticize the inventors of the IAT and state that they developed a test that reliably produces statistically significant results. This makes it easier for researchers to use the IAT for their own purposes and. Also, the more researchers publish results based on the IAT, the higher the motivation for justification of the tool and its results.

Why does the implicit prejudice still persist?

The authors of this article state that the IAT is very popular among psychologists, because it makes it easy to produce statistically significant effects. They state that even though there is a lot of criticism on the IAT, it is still very popular and this is due to ideological ideas, publication bias and the lack of clear, consensual score-keeping measures within the social psychology. The publication bias is about that it is favored to publish significant experimental results. The IAT studies are therefore easier to publish and popular among psychologists. According to the authors, the research domain of implicit prejudice is a good domain for experimenting with extraordinary science. They state that the rules for ‘success’ should be set ex ante, rather than allowing test-takers to post-hoc modify any pattern of results according to their preferred theories.

The application of statistics to science is not a neutral act. Textbook writers in the social sceiences have transformed rivaling statistical systems into an apparently monolithic method that could be used mechaniscaly. No scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects hypotheses; he rather gives his mind to each particular case in the light of his evidence and his ideas.

If statisticans agree on one thing it is that scientific inference should not be made mechanically. Good science requires both statistical tools and informed judgment about what model to construct, what hypotheses to test, and what tools to use. Many social scientist vote with their feet against an informed use of inferential statistics. A majority still computes p values, confindence intervals and a few calculate Bayes factors. Determining significance has become a surrogate for good research. This article is about the idol of a universal method of statistical inference.

Mindless statistical inference

In an internet study they asked participants if they felt a difference between heroism and altruism? The far majority felt so, and the authors computed a chi-squared test to find out whether the two numbers differed significantly. This is an illustration of the automatic use of statistical procedures, even when a statistical procedure really doesn't fit into the question. The idol of an automatic, universal method of inference, however, is not unique to p values or confidence intervals. It can also invade the Bayesian statistics.

The Idol of Universal Method of Inference

In this article they make three points:

1. There is no universal method of scientific inference, but, rather a toolbox of useful statistical methods. In the absence of a universal method, its followers worship surrogate idols, such as significant p values. The gap between the ideal and its surrogate rested on the wrong ideas people have regarding statistical inference. For instance that a p value of 1% indicates that there is a 99% chance of replication.
2. If the proclaimed 'Bayesian revolution' were to take place, the danger is that the idol of a universal method might survive in a new guise, proclaiming that all uncertainty can be reduced to subjective probabilities.
3. Statistical methods are not simply applied to a discipline; they change the discipline itself and vice versa.

In science and everyday life, statistical methods have changed whatever they touched. The most dramatic change brought about by statistics was the 'probabilistic revolution'. In the natural science, the term statistical began to refer to the nature of theories, not the evaluation of data.

How statistics changed theories: the probabilistic revolution

The probabilistic revolution upset the ideal of determinism shared by most European thinkers. It differs from other revolutions because it didn't replace any systems in its own field. But it did upset theories in other fields outside of mathemetics. The social sciences inspired the probabilistic revolution in physics. But the social and medical sciences were reluctant to abandon the ideal of simple, deterministic causes. The social theorists hesitated to think of probability as more than an error term in the equation observation = true value + error.

The term inference revolution refers to a change in scientific method that was instutionalized in psychology and in other social sciences. The qualifier inference indicates that the inference of a sample to population grew to be considered the most crucial part of research.

To understand how deeply the inference revolution changed the social sciences, it is helpful to realize that routine statistical tests, such as calculations of p values or other inferential statistics, are not common in the natural sciences.

The first known test of a null hypothesis was by Arbuthnott and is strikingly similar to the 'null ritual' that was instutionalized in the social sciences. He observed that the external accidents which males are subject do make a great havock of them, and that this loss exceeds far that of the other Sex. The first null hypothesis test impressed no one, but this does not say that statistical methods have played no role in the social sciences. To summarize, statistical inference played little role and Bayesian inference virtually none in research before roughly 1940. Automatic inference was unknown before the inferental revolution with the exception of the use of critical ratio (the ratio of the obtained difference to its standard deviation).

The Null Ritual

The most prominent creation of a seemingly universal inference method is the null ritual:

1. Set up a null hypothesis of 'no mean differences' or 'zero correlation'. Do not specify the predictions of your own research hypothesis.
2. Use 5% as a convention for rejecting the null. If significant, accept your research hypothesis. Report the result as p<.05>p<.01>, p.001, whichever comes next to the obtained p value.
3. Always perform this procedure.

In psychology, this ritual became institutionalized in currricula, editorials and professional associations. But the null ritual does not exist in statistics proper. Also the null ritual is often confused with the Fisher's thoery of null hypothesis testing. For example, it has become common to use the term NHST (null hypothesis significance testing) without distinguishing between the two. But contrary to what is suggested by that misleading term, level of significance has three meanings: (a) a mere convention, (b) the alpha level, or (c) the exact level of significance.

The three meanings of significance

The alpha level is the long-term relative frequency of mistakenly rejecting the hypothesis H1 if it is true, also known as the Type 1 error rate. The beta level is the long-term relative frequency of mistakenly rejecting hypothesis H2 if it is true, also known as the type 2 error rate or power- 1.

1. Set up two statistical hypotheses, H1 and H2 and decide on the alpha, beta and sample size before the experiment.
2. If the data falls into the rejection region of H1, accept H2; otherwise accept H1.
3. The usefulness of this procedure is limited among other situations were there is a conjunction of hypothese, where there is repeated sampling.

Fisher eventually refined his earlier position. The result was that a third definition of level of significance, alongside convention and alpha level.

1. Set up a statistcal null hypothesis. The null need not be a nil hypothesis.
2. Report the exact level of significance, do not use a conventional 5% level all the time.
3. Use this procedure only if you know little about the problem at hand.

The procedure of Fisher differs fundamentally from the null ritual. First, one should not automatically use the same level of significance, and second, one should not use this procedure for all problems. Step one of the ritual does contain the misinterpretation that null means 'nil' such as zero difference.

The problem of conflicting methods

When writers learned about Neyman-Pearson these writers had a problem; how should they deal with conflicting methods? The solution would have been to present a sort of toolbox of different approaches, but Guilford and Nunnally mixed the concepts and presented the muddle as a single, universal method. The idol of this universal method also left no place for Bayesian statistics.

Bayesianism and the new quest for a universal method

Fisher, Neyman and Pearson also have been victims of social scientists' desire for a single tool, a desire that produced a surrogate number for inferring what is good research. The potential danger of the Bayesian statistics lies in the subjective interpretation of probability, which sanctions its universal application to all situations of uncertainty.

The 'Bayesian revolution' had a slow start. To begin with, Bayes' paper was eventually published, but it was largely ignored by all scientists. Just as the null ritual had replaced the three interpretations of level of significance with one, the currently dominant version of Bayesianism does the same with the Bayesian pluralism, promoting a universal subjective interpretation instead. Probability was:

1. a relative frequency in the long run, such as in mortality tables used for calculating insurance premiums
2. a propensity, that is, the physical design of an object, such as that of a dice or a billiard table
3. a reasonable degree of subjective belief, such as in the attempts of courts to quantify the reliability of witness testimony.

In the essay of Bayes, his notion of probability is ambiguous and can be read in all three ways. With this ambiguity, however, is typical for his time in which the classical theory of probability reigned.

If probability is thought of as a relative frequency in the long run, it immediately becomes clear that Bayes' rule has a limited range of applications. Knight (economist) used the term risk for these two situations (i.e. probabilities that can be reliable measured in terms of frequency or propensity) as opposed to uncertainty. Subjective probability can be applied to situations of uncertainty and to singular events, such as the probability that Michael Jackson is still alive. There is now a new generation of Bayesians who believe that Bayesianism is the only game in town. They use the term Universal Bayes for the view that all uncertainties can or should be represented by subjective probabilities, that explicitly rejects the idea of Knight regarding the distinction between risk and uncertainty.

Risk versus uncertainty

What the universal Bayesians do not seem to realize is that in a theory Bayesianism can be optimal in a world of risk, but is of uncertain value when not all information is known or can be known or when probabilities have to be estiated form small, unreliable samples. One can also use plain common sense to see that complex optimization algorithms are unreliable in an uncertain world.

The automatic Bayes

As with the null ritual, the universal claim for Bayes' rule tends to go together with the automatic use. One version of the automatic Bayes has to do with the interpretation of the Bayes factors using the Jeffrey's scale. A second version of Automatic Bayes can be found in the heuristic-and-biases research program, that is widely taught in business education courses. But, in short, the automatic use of Bayes'rule is a dangerously beautiful idol. But Bayesianism is not reality, Bayesianism can't exist in the singular.

The statistical toolbox

The view of this article states that an alternative to these approaches is to think of the Universal and Automatic Bayes as forming a part of a larger toolbox. In this toolbox, the Bayes' rules has its value, but like any other tool, does not work for all problems.

How to change statistics?

Leibniz had a dream: to discover the calculus that could map all ideas into symbols. Such a universal calculus would also put an end to all scholarly bickering. But, nonetheless, this dream of Leibniz is still alive in social sciences today. The idea of surrogate science; from the mindless calculation of p values or Bayes factors to citation counts, is not entirely worthless. It fuels a steady stream of work of average quality and keeps researchers busy producing more of the same. But it also makes it harder for scientists to be innovative, risk taking and imaginative. Therefor surrogates also encourage cheating and incomplete or dishonest reporting. Would a Bayesian revolution lead to a better world? The answer depends on what the revolution might be. The real challenge here is to prevent the surrogates from taking over once again, such as when replacing routine significance tests with routine interpretations of the Bayes factors. So, Leibniz's beautiful dream of a universal calculus could easily turn into Bayes' nightmare.