The preregistration revolution – Nosek et al. - 2018 - Article

What is the aim of scientific progress?

Scientific progress is characterized by reducing uncertainty about nature. Models explaining prior and predicting future observations are constantly improved by reducing the prediction error. When the prediction error decreases, certainty about what will happen increases.

What is prediction and postdiction?

Scientists improve models by generating hypotheses based on existing observations, and testing them. We use general terms to capture the distinction – postdiction and prediction.

• Prediction: data used to confront the chance a hypothesis/prediction is wrong. Uses data to generate hypothesis about why something happened. Assess uncertainty by observing how predictions account for new data.
• Postdiction: data is known and hypothesis generated to explain why they occurred. Gathers data to test ideas about what will happen. Vital for discoveries of possibilities not yet considered.

Why is this distinction important?

Not knowing the difference can lead to overconfidence in post hoc explanations (postdictions) and increase the likelihood of believing that there is evidence for a finding when in fact there is no evidence. presenting postdictions as predictions can increase the attractiveness/publishability of findings by falsely reducing uncertainty – decreasing reproducibility.

Mistaking postdiction as prediction underestimates the uncertainty of outcomes, producing psychological overconfidence in resulting findings.

What are some mental constraints (biases) in distinguishing predictions and postdictions?

The dynamism of research and limits of human reasoning make it easy to mistake postdiction as prediction.

• Circular reasoning – generating a hypothesis based on observed data, then evaluating the validity of the hypothesis based on the same data.
• Hindsight bias (I-knew-it-all-along effect) – tending to see outcomes as more predictable after knowing data compared with before. Thinking you would have anticipated that explanation in advance. Common case – researcher’s prediction is vague enough that many outcomes can be rationalized.

How do novel findings lead to postdictions?

Novel (new findings), positive (finding an effect), clean (providing strong narrative) results are better for reward and for launching science into new domains of inquiry. They lead to postdictions because scientists are motivated to explain after the fact. If certain results are more rewarded than others, researchers are motivated to get results that are likely to be rewarded regardless of accuracy.

Lack of clarity between postdiction and prediction gives the chance to select, rationalize, and report tests that prioritize reward over accuracy.

How do standard tools of statistical inference assume prediction?

Null hypothesis significance testing (NHST) is made to test hypotheses (prediction). The prevalence of NHST and the P value implies that most research is prediction or that postdiction is often mistaken as prediction.

What is the garden of forking paths?

There are a many choices for analyzing data that can be made. If they’re made during analysis, observing data may make some paths more likely than others. In the end, it could be impossible to guess the paths that could’ve been chosen if the data were different or whether analytic decisions were influenced by certain biases.

How can preregistration distinguish prediction and postdiction?

Preregistration is committing to analytic steps without previous knowledge of research outcomes. Prediction is achieved because test selection isn’t influenced by observed data. Preregistration constrains how the data will be used to confront research questions.

• Inferences from preregistered analysis should be more replicable than those not preregistered – analysis choices can’t be influenced by motivation, memory, or reasoning biases.
• Preregistration can draw a line between pre-and-postdiction, preserving diagnosticity of NHST inference for prediction and clarifies the role of postdiction in generating possible explanations to test as future predictions.

Preregistration doesn’t favour prediction over postdiction, but aims to clarify which is which. In an ideal world preregistration would look like this: observation about the world > research questions > analysis plan > confront hypothesis > explore data for potential discoveries generating hypotheses after the fact > interesting postdictions converted to predictions or designing the next study and cycle repeats.

This ideal model is a simplification of how most research actually happens.

What are some challenges in preregistration?

Challenge 1: Changes to Procedure During Study Administration

Good plans can still be hard to achieve. Example: Jolene preregisters an experimental design using infants as participants. She plans to collect 100 observations but only gets 60. Some infants also fall asleep during the study, which she didn’t anticipate. Her preregistration analysis does not exclude sleeping babies. She can document changes to her preregistration without diminishing diagnosticity, and be transparent in reporting changes and explaining why. Most of the plan is still preserved and most deviations are transparently reported making it possible to assess their impact – benefit.

There’s an increased risk of bias with deviations from analysis plans after observing data, even when transparently reporting. With transparent reporting, observers can assess deviations and their rationale. The only way to achieve transparency is through preregistration.

Challenge 2: Discovery of Assumption Violations During Analysis

An example: Courtney discovers that the distribution of one of her variables has a ceiling effect and another is not normally distributed, violating the assumptions of her preregistered tests. Nevertheless, strategies are available to deal with contingencies in data-analytic methods without undermining diagnosticity. E.g. blinding, preregistering a decision tree, defining state etc.

Challenge 3: Data are Preexisting

An example: Ian uses data given by others to do his research. In most cases he does not know which variables/data are available to analyze until after data collection, making it difficult to preregister. The extent to which it is possible to test predictions on pretexting data depends on if decision analysis plan decisions are blind to the data. ‘Pure’ preregistration is possible if nobody has seen the data, e.g. an economist can make predictions of existing government data that hasn’t been released. But researchers that read summaries of reports or get advice on how to approach a data set by prior analysis might be influenced by their analysis.

Challenge 4: Longitudinal Studies and Large, Multivariate Datasets

An example: Lily leads a project making yearly observations of many variables over a 20-year period. Her group conducts dozens of investigations, Lily could not have preregistered the entire design and analysis plan for all future papers at the start.

Challenge 5: Many Experiments

An example: Natalie’s lab quickly gathers data, running multiple experiments a week. Preregistering every experiment seems burdensome for their efficient workflow. Teams that run many experiments usually do so in a paradigm where each experiment varies some key aspects of a common procedure. In this situation preregistration can be as efficient as the design of the experiments.

Challenge 6: A Program of Research

An example: Brandon’s research is high risk and most research outcomes are null results. Once in a while he gets a positive result with huge implications. Though he preregisters everything, he can’t be completely confidence in his statistical inference. There’s a chance of false positives. Another key element of preregistration – all outcomes of the analysis plan must be reported to avoid selective reporting.

Challenge 7: Few a Priori Expectations

An example: Matt doesn’t think that preregistration is useful to him because he thinks his research is discovery science. It’s usual to start research with few predictions. It’s less common for research to stay exploratory through a sequence of studies. If the data are used to formulate hypotheses instead of claiming evidence for the hypotheses, then the paper may be embracing post hoc explanations to open/test new areas of inquiry. But there are reasons to believe that sometimes postdiction is recast as prediction. In exploratory research, P values have unknown diagnosticity and using them can imply testing (prediction) rather than hypotheses generation (postdiction). Preserving the diagnosticity of P values means only reporting them when testing predictions.

Challenge 8: Competing Predictions

An example: Rusty and Melanie agree on the study design and analysis of their project but have competing predictions. This isn’t a challenge for preregistration. It has desirable characteristics that could lead to strong inference for favouring a theory. Prediction research can hold multiple predictions simultaneously.

Challenge 9: Narrative Inferences and Conclusions

An example: Alexandra preregistered her study, reporting all preregistered outcomes and distinguished the outcomes of tested pre-and-postdictions generated from the data. Some of her tests gave more interesting results than others and so her narrative, naturally, focused on the interesting ones. You can follow preregistration and still leverage chance in the interpretation of results. You could conduct 10 analyses, but the narrative of the paper could focus on two of them, increasing how the paper is applied and cited through inferential error. Can be solved statistically by e.g. applying alpha corrections so that positive narratives focus isn’t associated with inflation of false positives.

How can preregistration become the norm?

Culture today gives the means (biases and misuse), motive (to be punished), and opportunity (no prior commitments to predictions) for dysfunctional research practices. This is ow shifting to provide means, motive, and opportunity for robust practices through preregistration.

How can means be advanced for preregistration?

• A barrier to preregistration is insufficient/ineffective training of good statistical/methodological practices – online education modules available to facilitate learning.
• Research domains that require submission for ethics review for research on humans and animals should specify some of the methodology before doing the research.
• Thesis proposals for students often require comprehensive design and analysis plans that can easily become preregistrations.

How could motive be advanced for preregistration?

• Today there relatively weak incentives for research rigor and reproducibility, this is changing. Preregistration is required by US law for clinical trials and is necessary for publication.
• Journal funders are beginning to adopt expectations for preregistration, researchers’ behaviour is expected to follow.
• Some journals have come up with badges for preregistration as incentives to give credit for having preregistered with explicit designation on a published article.
• Incentives for preregistration in publishing. The preregistration challenge offers $1,000 awards to researchers that publish results of a preregistered study.
• Submitting their research question and methodology to journals for peer assessment before observing the outcomes.

How could opportunity be advanced for preregistration?

• Existing domain specific and general registries make it possible for researchers in any discipline to preregister their research.

Conclusion

Sometimes researchers use existing observations to generate ideas about how the world works – postdiction. Other times they have an idea about how the world works and make new observations to test if the idea is a reasonable explanation – prediction. To make confidence inferences, it’s important to know the difference and preregistration solves this challenge by making researchers state how they’ll analyze data before they observe it, allowing them to confront a prediction with the chance of being wrong.