False-positive psychology: Undiscovered flexibility in data collection and analysis allows presenting anything as significant - Simmons et al. - 2011 - Article

Introduction

A false positive is likely the most costly error that can be made in science. A false positive is the incorrect rejection of a null hypothesis.

Despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not.

Many researchers often stop collecting data on the basis of interim data analysis. Many researchers seem to believe that this practice exerts no more than a trivial influence on the false-positive rates.

Solutions for authors

The authors of this article offer six requiremets for authors as a solution to the problem of false-positive publications:

1. Before the collection of data begins, authors must decide the rule for terminating data collection and they should report this rule in the article.
2. At least 20 observations per cell must be collected by the author or else the author should provide a compelling cost-of-data-collection justification.
3. All variables collected in a study must be listed.
4. All experimental conditions must be reported, including failed manipulations.
5. If observations are eliminate, authors must also report what the statistical results are if those observations are included.
6. Authors must report the statistical results of the analysis without the covariate, if an analysis includes a covariate.

Guidelines for reviewers

The authors of this article also offer four guidelines for reviewers:

1. Reviewers must make sure that authors follow the requirements.
2. Reviewers should be more tolerant of imperfections in results.
3. Reviewers must make possible that authors are able to demonstrate that their results do not hinge on arbitrary analytic decisions.
4. Reviewers should require the authors to conduct an exact replication, if justifications of data collection or analysis are not compelling.

Conclusion

The solution offered does not go far enough in the sense that it does not lead to the disclosure of all degrees of freedom. It cannot reveal those arising from reporting only experiments that ‘work’ (i.e., the file-drawer problem).

The solution offered goes too far in the sense that it might prevent researchers from conducting exploratory research. This does not have to be the case if researchers are required to report exploratory research as exploratory research. This also does not have to be the case if researchers are required to complement it with confirmatory research consisting of exact replications of the design and analysis that ‘worked’ in the exploratory phase.

The authors considered a number of alternative ways to address the problem of reasearcher degrees of freedom. The following are considered and rejected:

• Correcting the alpha levels. A researched could consider adjusting the critical alpha level as a function of the number of researcher degrees of freedom employed in each study.
• Using Bayesian statistics. This approach has many virtues, it actually increases researcher degrees of freedom by offering new set of analyses and by requiring to make additional judgments on a case-by-case basis.
• Conceptual replications. They are misleading as a solution to the problem at hand, because they do not bind researchers to make the same analytic decisions across studies.
• Posting materials and data. This would impose too high a cost on readers and reviewers to examine the credibility of a particular claim.

The goals of researchers is to discover the truth, and not to publish as many articles as they can. For different reasons researchers could lose sight of this goal.