Scientific utopia II: restructuring incentives and practices to promote truth over publishability - Nosek, Spies & Motyl - 2012 - Article

What is the article about?

Publication influences hiring, salaries, promotions, tenures, grant decisions, and the evaluation of universities and departments. This leads researchers to want to publish as many articles as possible. The problem is that results are only publishable when they are novel (pushing the field in a new direction) and positive (happening in the expected direction). Novel and positive results are emphasized over negative results and replication. Science is about discovering new knowledge and thus it is not surprising there is a focus on novel findings. However, what happens is that incentives for publishable results can be at odds with incentives for accurate results, even though science's main motive is revealing the truth. In addition, the need for novel results makes it less likely that people run necessary replication studies, since there is less chance that these studies get published at all. The article suggests that this unwanted tendency can be tackled by making the more abstract goal of “getting it right” more interesting than the more clear-cut goal of “getting it published.”

What is the problem with a focus on novelty and positive results?

Findings are likely to be distorted when research is heavily geared towards novel and positive results, especially when combined with the fact that this affects a scientist's career and general well-being. Smaller departments might choose to lie and fake findings, because it would otherwise negatively influence other people's perceptions of them. Scientists might be tempted to engage in practices that increase their chances of publishing fake results, if circumstances allow them to do so. They might be likely to use designs, reporting and analysis strategies that increase the likelihood of positive results. This motivated reasoning can occur without a clear intention to do so. We are more likely to be convinced that our hypothesis is true, accepting it without criticism when it is confirmed and scrutinizing heavily when it is not. The focus on finding novel findings is misleading. Just because something has been published, does not mean it is true, especially if it has not been replicated. Because of the negative view of replication, it becomes difficult to get rid of a false finding once it has been published. It is striking that science, which differentiates itself from other approaches to acquiring knowledge (as it focuses on publicly disclosing the basis of evidence for a claim), has such a negative connotation of replication.

What increases the proportion of false results in the published literature?

Even though these practices can be justifiable at times, they also have the potential to increase the amount of published false results:

• Running many low-powered studies, rather than a few high-powered ones;

• Dismissing studies with negative results as pilot tests or methodologically flawed, but accepting studies with positive results as methodologically correct. This is all done in an environment that isn't highly critical

• Reporting only studies with positive results;

• Stopping with data collection as soon as an effect is obtained or the reverse: continuing data collection until an effect is obtained;

• Including multiple independent and/or dependent variables and only reporting on the set that generated results;

• Not specifying design and analysis models beforehand;

• Reporting a discovery as if it is the result of a confirmatory test;

• Not conducting a direct replication after finding an effect.

It is important that in cases where these practices are justified they should still be publicly disclosed, so that reviewers and other researchers can take it into account when evaluating the study and replicating the findings. The false positive rate has been estimated as high as 30%. Two different studies showed that in one only 11%, and in the other only 25% of published results could be replicated. The danger lies in the fact that, while results are not replicated, they still inspire researchers to conduct follow up studies expanding on elements of previously found findings, even though these findings might not be valid in the first place. For academic researchers, repercussions of having reported a false positive are minimal, the study is often not even retracted.

Which strategies are not sufficient to stop the high number of false results?

An overview of strategies:

• Conceptual replication: this involves deliberately changing the operationalization of some of the key elements of the design. This is very important in science when the constructs that are investigated cannot be directly observed. Because features of the original design are changed, this kind of replication can only be used as additional evidence for a found result, but it cannot actually disconfirm it. It might just not measure the original phenomenon. It can therefore easily contribute to confirmation bias.

• Seeing science as self-correcting: once published, there is no systematic procedure to determine the validity of the result. Therefore, false effects can remain to exist in scientific literature. And, retraction of confirmed false effects is rare.

• Journals devoted to replications or negative results: this kind of defines the journal as being of low importance: you can publish studies here that no other journal wants. It is not in the author's interest to publish in these kind of journals and thus it will do nothing against the motivation to obtain positive and novel results.

• Education campaigns: this topic has been a prominent topic of discussion for more than three decades and nothing has changed. The problem is that innovation is the key driver of progress and thus of science. Publishing has resource constraints. And thus the novel will be picked over the replication.

• Increasing expectations of reviewers to detect false results: of all procedures mentioned, this might be the most viable. Reviewers already have a critical mindset and are thus more likely to spot false results. And, some evaluation criteria could easily be implemented in a standard check list. However, the authors consider it to just be a part of the solution. Adding workload to voluntary work, to people who can also easily miss some details and only get to read the research summary, is not a full proof solution for detecting all false positives.

• Raising the barrier for publication, by requiring reports to include replications of new findings. One problem with this is that it further delays the already long during and difficult process of publication, thereby halting the progress of science. In addition, it could reduce risk taking and innovation, especially when resource demands are high.

Which strategies will promote the accumulation of knowledge?

Scientists are motivated by accuracy, and "getting it right" has better long-term consequences than just getting published. However, getting it published has immediate, palpable and concrete advantages, whereas getting it right is an abstract, long-run goal. The authors suggest that interventions should make sure that getting it right has more concrete and direct advantages in order to make it more competitive with the goal of simply getting published. They suggest the following strategies for that:

• Promoting and rewarding paradigm-driven research. This type of research gathers knowledge by systematically altering a procedure to investigate a hypothesis, rather than varying many features of the methodology. Hereby, a research can include replication and extension in one design. It combines replication with new knowledge, because of the systematic change in procedures. The most important risk with this method is that research questions can become more about the method itself than the theory that the method is intended to address. Using one methodology for a theoretical question cannot distinguish between specific features of that method from the actually phenomenon being investigated. Therefore, this research method should be combined with conceptual replications.

• Checklists for authors, reviewers and editors. Checklists are easy to implement and reduce additional workload on authors, reviewers and editors by predetermining what they should be on the lookout for. Checklists are necessary because key information is left out rather frequently.

• Changing mindsets. There is some evidence that even though some degree of publishing productivity gets you into a pool of shortlisted candidates, other factors are more important for actually getting the job.

• Metrics that identify what should be replicated. The amount of resources needed to replicate every study would undermine innovation. A solution is to develop metrics for identifying replication value (RV): which effects should be replicated? Currently a method is being developed that determines the RV value based on the citation impact and the precision of existing evidence of the effect.

• Crowd sourcing replication efforts. By sharing data efforts among multiple laboratories the resource burden of replications becomes manageable.

• Peer review standards emphasize soundness over importance of research. Even though the focus lies on “important” ideas, page limits become less and less of an issue in the digital age. For instance, PLoS ONE is an article where reviews are solely focused on the soundness of the studies submitted, and it has shown an amazing growth with an impressive impact factor.

• Making it easier to publish. Let authors themselves decide whether they want to publish or not, and then include peer-review after publication as an evaluation mechanism. This way, publishing becomes less of an incentive. Also, it will shift the peer-review process from focusing on whether the manuscript should be published to whether the study was sound and how it could be improved. This would also change the mindset that publication is the end of the research process. In the current state, publishing is more rewarding instead than knowledge building. In the new state, the value of publishing is reduced and thereby the comparative value of having an impact on other research increases.

• Open access to data, materials and workflow. All the aforementioned strategies will make it more rewarding to contribute to knowledge accumulation and less rewarding to publish. However, it does not contribute to openness of the research conducted. In current research practice, openness only happens through the journal article. However, openness is of outmost importance to science: it increases accountability, gives the science community the chance to correct mistakes and makes it easier to share and make use of materials, methods and data. Three areas of science are very closed: data, methods and tools.

Why is open data important?

Data can be used to confirm made conclusions, to reanalyze them with new techniques, to aggregate data across multiple studies and it increases the credibility of science. Errors cannot only occurs in statistics, but also in data coding, cleaning, analysis and results reporting. These errors cannot be detected in the research summary that is published. The importance of open data is confirmed by a study that found that reluctance to share data was associated with weaker evidence against the null hypothesis and more statistical errors, in particular those that made a difference for whether significance was found or not. Barriers to sharing open data are additional formatting requested of the researchers and a reduction in confidentiality, by sharing the data with more people than just the research team and storing the data for a prolonged period of time. Also, there are legal concerns and data sets might need to be used for multiple studies. Therefore, even though a default to open data would be helpful, guidelines need to be build in for when keeping data closed is justified.

Why are open methods and tools important?

They also allow confirmation and improvement of previous research and it makes progress in used methods easier. It facilitates replication and paradigm-driven research. Unfortunately, research reports often include too little detail to allow for replication, because some details that are of importance do not necessarily seem important to the researcher (e.g. time of day data collection took place). Sharing all details makes replications more accurate in similarity to the original study and it reduces the time lost in trying to copy previously used methods. An added benefit for the research is that contributing data or methods that are used in multiple studies, increases the reputation of the owner of the resources.

Why is an open workflow important?

Public documentation of the whole research process makes it easier to detect practices that bias the accuracy of what is reported. For instance, by registering random control trials, these trials can be compared to those that are actually published, making it possible to detect discrepancies. This can help us determine how findings came about: was it a discovery in the course of the research or was it predicted beforehand. Even though not everybody might be interested in all these details, reporting every step in the process reduces bias in the end reporting and it also makes it easier to replicate or extend previous studies.