“Nosek, Spies, & Motyl (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability.” - Article summary

There is a publication bias in research. Mostly positive results – that is significant – are being published and non-significant results are often not published. This leads to researchers only trying to find significant results instead of genuine effects or non-effects.

Researchers are more likely to find significant results because of flexible analysis options (1), the confirmation bias (2) and the drive to find significant results (3). It is possible to find significant results everywhere if you search hard enough. Direct replications do not occur often in research and are almost never published.

There are several practices that can increase publishability but can decrease the validity of the results:

1. Leveraging chance by running many low-powered studies instead of a few high-powered ones.
2. Uncritically dismissing “failed” studies as pilot studies but uncritically accepting “successful” studies.
3. Selectively reporting studies with positive results and not studies with negative results (cherry picking).
4. Stopping data collection as soon as a reliable effect is obtained.
5. Continuing data collection until a reliable effect is obtained.
6. Including multiple independent and/or dependent variable and reporting the subset that “worked”.
7. Maintaining flexibility in design and analytical models (attempt to exclude data)
8. Reporting a discovery as if it had been the result of a confirmatory test
9. Not doing a direct replication once a reliable effect is obtained

Conceptual replication involves deliberately changing the operationalization of the key elements of the design such as the independent variable, dependent variable or both. Demonstrating the same effects with multiple operationalizations provides confidence in its conceptual interpretation. Conceptual replication is not an effective replacement for direct replication.

The peer review process offers a way to detect false results. Not publishing articles without replications would also be effective in lessening the publication bias and the number of false results in science. This could also be ineffective, because it could reduce the number of innovative research as scientists might ‘play it on safe’.

Paradigm-driven research can be used to both confirm and disconfirm prior results. It accumulates knowledge by systematically altering a procedure to investigate a theory or a research question. This includes both replication and extension of the research. One pitfall of paradigm-driven research is that the research could become about the methodology, instead of the theory. Conceptual replication prevents this.

Check-lists could provide effective when conducting research as this prevents information from being left out. A metric that determines what is worth replicating could also prove to be effective. Resource constraints to replicate some findings could be overcome by using crowdfunding. Peer reviewers judge whether a finding is important enough to be published, but they are not always capable of making that judgement.

Another solution to the publication bias and the number of false results in science is to shift the attention from publishing – make publishing trivial – to the evaluation of the research.

The solution to the previously described problems would be open data (1), open materials (2) and open workflow (3).

Open data refers to sharing the data which allows other researchers to confirm the data. Making data open increases the likelihood of finding and correcting errors and ultimately improving reported results. Data openness can be a problem when the data cannot ensure anonymity of the participant’s identity or when the data is going to be used for a lot of research projects and the data collection process is intense.

Open methods and materials can have the same effect as open data and can also facilitate progress in reuse, adaptation and extension for new research. The availability of the materials and methods will also speed up productivity by eliminating the need to reinvent or recreate them.

Open workflow refers to openness of the research process. This could include preregistering the research process, what statistical analyses will be used and so on. This makes it more difficult to hide undesirable results.