Seminar 2: Open Science (ARMS, Utrecht University)

*These notes do not include the introduction to R*

HARKing: changing or creating the hypothesis after seeing the results.

P-hacking: researchers collect or select data or statistical analyses until nonsignificant results become significant.

Selective reporting: only reporting significant results, and not reporting the non-significant results.

Publication bias: only articles with significant effects are published.

So the problem is: the literature is not representative for the population.

What can we do to avoid these biases/to make ‘great results’ less important? --> Registered reports: when you use pre-registration it is on forehand clear what you’re going to do and what you’re interested in, in a manner that is verifiable by others. Four central aspects of the Registered Reports Model are:

• Researchers decide hypotheses, study procedures and main analyses before data collection
• Part of the peer review process takes place before studies are conducted
• Passing this stage of review virtually guarantees publication
• Original studies and high-value replications are welcome

Methods of pre-registration include:

• Sample (size)
• Design, variables
• Measures
• Exclusion criteria
• Analysis plan

How does it work? Authors submit the STAGE 1 manuscript, stage 1 peer review takes place and if the reviews are positive, the journal offers in-principle acceptance (IPA) regardless of the study outcome.

The advantages of Registered Reports:

--> For the scientific community:

• Rigorous review of theory and methods
• Eliminates publication bias and reporting bias
• Increases the reproducibility of science

--> For scientists

• Keeping track of what you did and why
• Peer review when it is most helpful
• Publication guaranteed regardless of the results

Common misconceptions:

1. ‘Pre-registration prevents the exploration of your data/creativity’. This is not the case. It allows for exploratory science, it simply prevents reporting exploratory analyses as confirmatory.

2. ‘Pre-registration does not allow for making changes and I cannot predict what will happen’. This is not correct. It allows for these things, as long as you report your deviation from your pre-registered plan.

Questions? Let me know in the contribution section!

Follow me for more summaries on statistics!