What is the scientific method?

The scientific method is an empirical and sceptical approach to asking questions and obtaining answers.

Good habits for a researcher include enthusiasm, open-mindedness, common sense, role-taking ability, inventiveness, confidence in own judgement, consistency and care for details, ability to communicate, and being honest.

What is the issue in psychological research?

Evidence shows a deficiency in most social sciences to match the growth and theoretical integration that characterizes the history of more successful scientific disciplines. Meehl presupposes that, with certain exceptions, theories in ‘soft areas’ of psychology tend to go through periods of initial enthusiasm leading to large amounts of empirical investigation with ambiguous overall results. This is followed by various kinds of amendment and the generation of ad hoc hypotheses. In the long run, experiments lose interest instead of theories actually being falsified. “They never die; they just slowly fade away.”

This discussion constitutes research that shares three properties: (a) theories in ‘soft areas’, (b) data correlational, and (c) positive findings consisting of refuting the null hypothesis. Soft areas could include counselling, personality theory, and social psychology.

What is Meehl’s thesis?

“Null hypothesis testing of correlational predictions from weak substantive theories in soft psychology is subject to the influence of ten obfuscating factors whose effects are usually (1) sizeable, (2) opposed, (3) variable, and (4) unknown. The net effect of these ten obfuscating influences is that the usual research literature review is almost uninterpretable.”

The derivation of observational conditional

Theories, taken with a statement of conditions, entail relations between single observations (particulars). The derivation of a prediction about observational facts involves a conjunction of several premises.

• Derivation of observational conditional: T – A1 – A2 – Cp – Cn à (O1 É O2)
  • T: substantive theory of interest
  • A1, A2…: one or more auxiliary theories which aren’t the main focus of the investigator’s interest.
  • Cp: ceteris paribus clause, a negative statement not formulated with concrete content like an auxiliary but states ‘other things being equal’.
  • Cn: statements about the experimental ‘conditions’.
  • O1 & O2: observational statements.

Meehl’s ten obfuscating factors

The combined operation of the ten factors makes it impossible to tell what a score of statistical significance in research proves about a theory’s plausibility.

1. Loose derivation chain

There are very few tight derivation chains in most areas of psychology and almost none in soft psychology. Some parts of the chain are well explained, while others are vague or not explicitly stated. Meehl states that if your observation 2 doesn’t follow observation 1 as expected, you won’t know why it happened and which part of the derivation chain was problematic.

2. Problematic auxiliary theories

Auxiliary theories are explicitly stated. Sometimes it happens in soft psychology that each auxiliary theory is almost as problematic as the main theory we’re testing. When there are multiple problematic auxiliary theories, the joint probability that they have could be lower than the prior probability of the theory of interest. We intended to investigate T, but the logical structure leaves us not knowing whether the prediction failed because our T was false or because one or more of the conjoined statements (A1, A2, Cp, or Cn) were false. – this reasoning applies to sections 3 and 4 as well.

3. Problematic ceteris paribus clause

Ceteris paribus means ‘everything else being equal’, but everything else isn’t always equal. This clause means that when we test a theory by showing a statistical relationship, while individuals may vary in certain factors that we have not controlled but allowed to vary, the alleged causal influence doesn’t have a significant additional effect operating in a direction opposed to our theory. There is usually a problem of control in the study. There are some situations where we can’t or don’t know how to control for certain factors, which could to some extent counteract the induced state we’re trying to test.

4. Experimenter error

Experimenter mistakes in manipulation. Includes errors due to biases, expectancies etc. on the part of the experimenter (or others involved on the input side of a testing process like research assistants). The experimenter may be enthusiastic about their theory and, without consciously intending, slant results into their favour, e.g. by the way they interact with their subjects/keenness.

5. Inadequate statistical power

Most tests in psychology can be expected to fail because of insufficient statistical power (usually due to small sample sizes) to detect a ‘real effect’.

6. Crud factor

In social sciences, everything correlates to some extent with everything else. Any measured trait is some function of a list of causal factors in the genes and history of the individual. Genetic and environmental factors are also known from empirical research to be correlated. 

7. Pilot study

A pilot study essentially duplicates a main study. A true pilot study is a mini version of a main study with a few minor improvements perhaps. Pilot studies have two aims, firstly they try to see whether an effect exists or not (to then decide whether to pursue the research to a larger extent). Secondly, they try to get a rough idea of the relationship between a mean difference and the approximate variability as a basis for inferring the number of cases that would be necessary to achieve statistical significance. If nothing significant is found, the pilot will be discarded and not published (because no effect = evidence against a theory). This means there’s a large collection of data missing from the collective understanding of the human psyche.

8. Selective bias in submitting reports

A researcher is more likely to submit a study if significant results were found. This comes partly from thinking that a null result ‘doesn’t prove much’. This creates bias in the literature available to the population.

9. Selective editorial bias

People who have been editors or referees say the same thing as investigators: they’re somewhat more inclined to publish a clear finding of a refuted null hypothesis than one that simply fails to show a trend. There’s a failure to report everything. Non-significant results are deemed to say that ‘there’s nothing here’.

10. Detached validation claim for psychometric instruments

Testing should be done with valid instruments. Sometimes, theories and instruments are validated at the same time. This isn’t good practice because you should establish validity of an instrument before you use it.

Introduction

Reproducibility is a core principle of scientific progress. Scientific claims should get credibility by the replicability of their evidence instead of the status of their originator. Debates about transparency of methodology and claims are meaningless if the evidence being debated is not reproducible.

What is replication?

Direct replication tries to recreate conditions believed necessary to obtain a previously observed finding and is a way to establish reproducibility of a finding with new data. A direct replication may not achieve the original result for many reasons:

• Known or unknown differences between the replication and original study could moderate the size of an observed effect.
• Original result could have been a false positive.
• The replication could produce a false negative.

False positives and false negatives provide misleading information about effects, and failure to identify the necessary conditions to reproduce a finding shows and incomplete theoretical understanding. Direct replication provides the chance to assess and improve reproducibility.  

Method & Results

The authors came up with a protocol for selecting and conducting high-quality replications. Collaborators joined the project, selected study for replication from available studies in the sampling frame, and were guided through the replication process.

The Open Science Collaboration conducted 100 replications (270 contributing authors) of studies published in three psychological journals using high-powered designs and original materials when possible. Through consulting original authors, obtaining original materials, and internal review, replications kept high fidelity to the original designs.

Here, they chose to evaluate replication success using significance and P values, effect sizes, subjective assessments of replication teams, and meta-analysis of effect sizes. The mean effect size of the replication effects was half the size of the mean effect size of the original effects, substantial decrease.

• 97% of original studies had significant results (P
• 36% of replications had significant results.
• 47% of original effect sizes were in the 95% confidence interval of the replication effect size.
• 39% of effects were subjectively rated to have replicated the original result.
• Assuming no bias in original results, combining original and replication results left 68% with statistical significant effects.

Tests suggest that replication success was better predicted by the strength of the original evidence than by characteristics of the original and replication teams.

Does successful replication mean theoretical understanding is correct?

It would be too easy to conclude that. Direct replication mainly provides evidence for the reliability of a result. Understanding is achieved through many, diverse investigations that give converging support for a theoretical interpretation and rules out alternative explanation.

Does failure to replicate mean the original evidence was a false positive?

It would also be too easy to conclude this. Replications can fail if the methodology differs from the original ways that interfere with observing the effect. The Open Science Collaboration conducted replications designed to minimize a priori reasons to expect different results by using original materials, contacting original authors for review of designs, and conducting internal reviews. However, unanticipated factors in the sample, setting, or procedure could have altered the observed effect sizes.

How is reproducibility influenced by publication bias?

There are indications of cultural practices in scientific communication that could be responsible for the observed results. The combination of low power research designs and publication bias produce a literature with upwardly biased effect sizes. This anticipates that replication effect sizes would regularly be smaller than the original studies. This isn’t because of differences in implementation but because the original effect sizes are influenced by publication/reporting bias and the replications aren’t.

Consistent with this expectation, most replication effects were smaller than the originals, and reproducibility success correlated with indicators of the strength of the initial evidence, like lower original P values and larger effect sizes. This suggests that publication, selection, and report biases are possible explanations for the difference in original and replication effects. The replication studies reduced these biases because replication preregistration and pre-analysis plans ensured confirmatory tests and reporting of all results.

Furthermore, repeated replication efforts that fail to identify conditions of the original finding can be observed reliably and could reduce confidence in the original finding.

Conclusion

The five indicators examined to describe replication success are not the only ways to evaluate reproducibility. But the results offer a clear, collective conclusion: a large portion of replications produced weaker evidence for original findings despite using materials provided by the original authors, internal reviews, and high statistical power to detect the original effect sizes.

Correlational evidence is consistent with the conclusion that variation in the strength of initial evidence was more predictive of replication success than variation in the characteristics of the teams conducting the research.

Reproducibility is not well understood because of the incentives for individual scientists to prioritize novelty over replication. Innovation is an engine of discovery and is vital for a productive, effective scientific interpose. But journal reviewers and editors might dismiss a new test of a published idea as unoriginal. Innovation shows that paths are possible, replication shows that paths are likely, progress relies on both. Scientific progress is a process of uncertainty reduction that can only succeed if science remains sceptical of its explanatory claims.

What is the aim of scientific progress?

Scientific progress is characterized by reducing uncertainty about nature. Models explaining prior and predicting future observations are constantly improved by reducing the prediction error. When the prediction error decreases, certainty about what will happen increases.

What is prediction and postdiction?

Scientists improve models by generating hypotheses based on existing observations, and testing them. We use general terms to capture the distinction – postdiction and prediction.

• Prediction: data used to confront the chance a hypothesis/prediction is wrong. Uses data to generate hypothesis about why something happened. Assess uncertainty by observing how predictions account for new data.
• Postdiction: data is known and hypothesis generated to explain why they occurred. Gathers data to test ideas about what will happen. Vital for discoveries of possibilities not yet considered.

Why is this distinction important?

Not knowing the difference can lead to overconfidence in post hoc explanations (postdictions) and increase the likelihood of believing that there is evidence for a finding when in fact there is no evidence. presenting postdictions as predictions can increase the attractiveness/publishability of findings by falsely reducing uncertainty – decreasing reproducibility.

Mistaking postdiction as prediction underestimates the uncertainty of outcomes, producing psychological overconfidence in resulting findings.

What are some mental constraints (biases) in distinguishing predictions and postdictions?

The dynamism of research and limits of human reasoning make it easy to mistake postdiction as prediction.

• Circular reasoning – generating a hypothesis based on observed data, then evaluating the validity of the hypothesis based on the same data.
• Hindsight bias (I-knew-it-all-along effect) – tending to see outcomes as more predictable after knowing data compared with before. Thinking you would have anticipated that explanation in advance. Common case – researcher’s prediction is vague enough that many outcomes can be rationalized.

How do novel findings lead to postdictions?

Novel (new findings), positive (finding an effect), clean (providing strong narrative) results are better for reward and for launching science into new domains of inquiry. They lead to postdictions because scientists are motivated to explain after the fact. If certain results are more rewarded than others, researchers are motivated to get results that are likely to be rewarded regardless of accuracy.

Lack of clarity between postdiction and prediction gives the chance to select, rationalize, and report tests that prioritize reward over accuracy.

How do standard tools of statistical inference assume prediction?

Null hypothesis significance testing (NHST) is made to test hypotheses (prediction). The prevalence of NHST and the P value implies that most research is prediction or that postdiction is often mistaken as prediction.

What is the garden of forking paths?

There are a many choices for analyzing data that can be made. If they’re made during analysis, observing data may make some paths more likely than others. In the end, it could be impossible to guess the paths that could’ve been chosen if the data were different or whether analytic decisions were influenced by certain biases.

How can preregistration distinguish prediction and postdiction?

Preregistration is committing to analytic steps without previous knowledge of research outcomes. Prediction is achieved because test selection isn’t influenced by observed data. Preregistration constrains how the data will be used to confront research questions.

• Inferences from preregistered analysis should be more replicable than those not preregistered – analysis choices can’t be influenced by motivation, memory, or reasoning biases.
• Preregistration can draw a line between pre-and-postdiction, preserving diagnosticity of NHST inference for prediction and clarifies the role of postdiction in generating possible explanations to test as future predictions.

Preregistration doesn’t favour prediction over postdiction, but aims to clarify which is which. In an ideal world preregistration would look like this: observation about the world > research questions > analysis plan > confront hypothesis > explore data for potential discoveries generating hypotheses after the fact > interesting postdictions converted to predictions or designing the next study and cycle repeats.

This ideal model is a simplification of how most research actually happens.

What are some challenges in preregistration?

Challenge 1: Changes to Procedure During Study Administration

Good plans can still be hard to achieve. Example: Jolene preregisters an experimental design using infants as participants. She plans to collect 100 observations but only gets 60. Some infants also fall asleep during the study, which she didn’t anticipate. Her preregistration analysis does not exclude sleeping babies. She can document changes to her preregistration without diminishing diagnosticity, and be transparent in reporting changes and explaining why. Most of the plan is still preserved and most deviations are transparently reported making it possible to assess their impact – benefit.

There’s an increased risk of bias with deviations from analysis plans after observing data, even when transparently reporting. With transparent reporting, observers can assess deviations and their rationale. The only way to achieve transparency is through preregistration.

Challenge 2: Discovery of Assumption Violations During Analysis

An example: Courtney discovers that the distribution of one of her variables has a ceiling effect and another is not normally distributed, violating the assumptions of her preregistered tests. Nevertheless, strategies are available to deal with contingencies in data-analytic methods without undermining diagnosticity. E.g. blinding, preregistering a decision tree, defining state etc.

Challenge 3: Data are Preexisting

An example: Ian uses data given by others to do his research. In most cases he does not know which variables/data are available to analyze until after data collection, making it difficult to preregister. The extent to which it is possible to test predictions on pretexting data depends on if decision analysis plan decisions are blind to the data. ‘Pure’ preregistration is possible if nobody has seen the data, e.g. an economist can make predictions of existing government data that hasn’t been released. But researchers that read summaries of reports or get advice on how to approach a data set by prior analysis might be influenced by their analysis.

Challenge 4: Longitudinal Studies and Large, Multivariate Datasets

An example: Lily leads a project making yearly observations of many variables over a 20-year period. Her group conducts dozens of investigations, Lily could not have preregistered the entire design and analysis plan for all future papers at the start.

Challenge 5: Many Experiments

An example: Natalie’s lab quickly gathers data, running multiple experiments a week. Preregistering every experiment seems burdensome for their efficient workflow. Teams that run many experiments usually do so in a paradigm where each experiment varies some key aspects of a common procedure. In this situation preregistration can be as efficient as the design of the experiments.

Challenge 6: A Program of Research

An example: Brandon’s research is high risk and most research outcomes are null results. Once in a while he gets a positive result with huge implications. Though he preregisters everything, he can’t be completely confidence in his statistical inference. There’s a chance of false positives. Another key element of preregistration – all outcomes of the analysis plan must be reported to avoid selective reporting.

Challenge 7: Few a Priori Expectations

An example: Matt doesn’t think that preregistration is useful to him because he thinks his research is discovery science. It’s usual to start research with few predictions. It’s less common for research to stay exploratory through a sequence of studies. If the data are used to formulate hypotheses instead of claiming evidence for the hypotheses, then the paper may be embracing post hoc explanations to open/test new areas of inquiry. But there are reasons to believe that sometimes postdiction is recast as prediction. In exploratory research, P values have unknown diagnosticity and using them can imply testing (prediction) rather than hypotheses generation (postdiction). Preserving the diagnosticity of P values means only reporting them when testing predictions.

Challenge 8: Competing Predictions

An example: Rusty and Melanie agree on the study design and analysis of their project but have competing predictions. This isn’t a challenge for preregistration. It has desirable characteristics that could lead to strong inference for favouring a theory. Prediction research can hold multiple predictions simultaneously.

Challenge 9: Narrative Inferences and Conclusions

An example: Alexandra preregistered her study, reporting all preregistered outcomes and distinguished the outcomes of tested pre-and-postdictions generated from the data. Some of her tests gave more interesting results than others and so her narrative, naturally, focused on the interesting ones. You can follow preregistration and still leverage chance in the interpretation of results. You could conduct 10 analyses, but the narrative of the paper could focus on two of them, increasing how the paper is applied and cited through inferential error. Can be solved statistically by e.g. applying alpha corrections so that positive narratives focus isn’t associated with inflation of false positives.

How can preregistration become the norm?

Culture today gives the means (biases and misuse), motive (to be punished), and opportunity (no prior commitments to predictions) for dysfunctional research practices. This is ow shifting to provide means, motive, and opportunity for robust practices through preregistration.

How can means be advanced for preregistration?

• A barrier to preregistration is insufficient/ineffective training of good statistical/methodological practices – online education modules available to facilitate learning.
• Research domains that require submission for ethics review for research on humans and animals should specify some of the methodology before doing the research.
• Thesis proposals for students often require comprehensive design and analysis plans that can easily become preregistrations.

How could motive be advanced for preregistration?

• Today there relatively weak incentives for research rigor and reproducibility, this is changing. Preregistration is required by US law for clinical trials and is necessary for publication.
• Journal funders are beginning to adopt expectations for preregistration, researchers’ behaviour is expected to follow.
• Some journals have come up with badges for preregistration as incentives to give credit for having preregistered with explicit designation on a published article.
• Incentives for preregistration in publishing. The preregistration challenge offers $1,000 awards to researchers that publish results of a preregistered study.
• Submitting their research question and methodology to journals for peer assessment before observing the outcomes.

How could opportunity be advanced for preregistration?

• Existing domain specific and general registries make it possible for researchers in any discipline to preregister their research.

Conclusion

Sometimes researchers use existing observations to generate ideas about how the world works – postdiction. Other times they have an idea about how the world works and make new observations to test if the idea is a reasonable explanation – prediction. To make confidence inferences, it’s important to know the difference and preregistration solves this challenge by making researchers state how they’ll analyze data before they observe it, allowing them to confront a prediction with the chance of being wrong.

Open science means doing scientific research in a transparent way, and making results available to everyone.

What is the problem?

Right now, open science is seen as optional, e.g. open access to articles is offered at an extra cost. Imagine the opposite (e.g. having to pay to make your work closed).

The ‘mobile phone paradox’- mobile phones are a world changing invention allowing people to connect from wherever they are. But sometimes it does not work because of signal issues. Should we not have invented mobile phones because they don’t work sometimes? No. The same is true for open science, it won’t always work but it’s the right thing to do.

There are six commonly accepted pillars of open science.

What is open data?

Releasing raw and processed data from our experiments, allowing others to analyze it unrestrictedly. The author thinks all raw data generated throughout your experiment should be released (especially discarded data), at least enough to completely regenerate the analysis you did (replicability. If the human genome project had only published the ‘interesting data’, many scientific discoveries would have been delayed.

Moral argument to disclose all data: data doesn’t belong to the scientists’; they belong to the funder (e.g. taxpayers). Datasets should be freely available to those who funded them.

What is open access?

A model where papers are available for anyone to read without paying, and that license allows secondary use like text-mining. It is immoral to expect those funding the research to pay to have access to the results.

What is open methodology?

A methodology that has been described in enough detail to allow researchers to repeat and apply the work elsewhere. A main reason we publish is so that others can learn from what we have done, showing how you carried out an experiment is essential to that.

What is open sourcing?

Refers to open and free access to the blueprint of a product. If you use software in your experiment, the source code should be available to read. It is part of the methods section and is the easiest part to share (online). Software should drive the open-science movement.

What is open peer-review?

It is about transforming the peer review process, making it collaborative between authors and reviewers through constructive criticism. Part of it is to remove anonymity. Evidence showed open review to do no damage to the quality of peer reviews.

What is open education?

Refers to the open and free availability of open resources. You can still charge for education, but resources used to educate should be made freely available.

The article discusses scientific and ethical issues relevant to conducting psychological research. Looking at considerations of research design, procedures, and recruitment of human participants.

What are issues of design?

A safe research proposal can be ethically questionable because of design issues. For example: research hypothesizing private schools to improve children’s intellectual functioning more than public schools. This design does not allow reasonable causal inference because there is no randomization or consideration of other hypotheses. Research is questionable if:

• Participants’ time is taken from more profitable experiences/treatments.
• Poor quality design leads to unwarranted/inaccurate conclusions possibly harming the society funding the research.
• Giving money and time to poor quality research keeps resources from better science.

What are issues of recruitment?

Hyperclaiming: telling prospective participants (+ granting agencies, colleagues etc.) that research is likely to achieve goals that it is unlikely to achieve. Colleagues and administrators can evaluate our claims fairly but our participants cannot. We should be honest with them about the realistic goals of the study.

Causism: tendency to imply a causal relationship where it has not been established.

Characteristics of causism:

• No appropriate evidential base.
• Presence of language implying cause (‘the effect of’, ‘as a result of’ etc.) where appropriate language would be ‘was related to’ or ‘could be inferred from’.
• Self-serving benefits to the causist because it makes the result appear more important than it really is.

If the causist is unaware of the causism, it reflects poor scientific training. If they are aware of it, reflects unethical misrepresentation and deception.

A description of a proposed research study using causal language represents an unfair recruitment device used to increase potential participation rates.

How does bad science make for bad ethics?

The author proposes that institutional review boards should consider the technical scientific competence of investigators whose proposals they evaluate. Poor quality research can make for poor quality education. Asking a participant to participate in bad research increases the likelihood of them acquiring misconceptions about the nature of science and psychology rather than benefitting educationally.

What are costs and utilities?

When presented questionable research proposals, investigators/review boards employ a cost-utility analysis where the costs of doing a study (time, money, negative effects on participants) are evaluated against utilities (benefits to participants, science, the world etc.). High quality studies/studies with important topics have a utility outweighing the costs. But it is hard to decide if a study should be done when the costs and benefits are equal. However, the costs for failing to do research also should be evaluated – focuses on the benefit for future generations or participants. Example: if people can receive free care during a study that otherwise they couldn’t afford, is it ethical to not conduct this research?

What does data dropping entail?

The goal to have more support for your hypothesis:

1. Outlier rejection: it is likely that researchers drop outliers that are inconsistent with their hypothesis than those falling in line with their hypothesis. Outlier rejection should be reported. Results with outlies should be reported if you decide to drop them.
2. Subject selection: different type of data dropping. Subset of the data is not included in the analysis. Even if there may be good technical reasons, there are still ethical issues like when just subsets not supporting the researcher’s hypothesis are dropped. If you drop subsets, readers should be informed about it and what the results were. Similar considerations apply when results for one or more variables are not reported.

Is data exploitation always bad?

This issue has subtler ethical implications. We are taught that it is improper to snoop around in our data (analyze and reanalyze). It makes for bad science because while snooping affects p values, it is likely to turn up something new. It makes for bad ethics because data are expensive in time, effort, and money and looking further into data may turn up something you may not have found otherwise.

The author says if the research is wroth conducting, it is also worth taking a closer look at it. Replications are needed anyway whether you snoop or not. Bonferroni adjustments can help with the significant p-value that you may find after exploiting your data.

Can meta-analysis be used as an ethical imperative?

Meta-analyses are a set of concepts and procedures used to summarize any domain of research. We see them as more accurate, comprehensive, and statistically powerful compared to traditional literatures review because they have more information. This leads to:

• More accurate estimates of effect sizes and relationships.
• More accurate estimates of overall significance levels of the research domain.
• More useful information about the variables moderating the magnitude of the effect.
• Increase of utilities: time, effort, costs are all more justified when datasets are in a meta-analysis because we can learn more from our data.
• Ethical implications of not doing meta-analyses: failing to employ met-analytic procedures means you lose the opportunity to make use of past research.

Meta-analyses try to explain the variation in effect sizes from different studies. It seems to no longer be acceptable to fund research resolving a controversy unless an investigator as already done a meta-analysis to decide if there really is a controversy.

Pseudocontroversies: meta-analysis resolves controversies because it eliminates two common problems in evaluating replications:

1. When failing to get a significant effect in the replication study, we fail to replicate – failure to replicate is measured by the size of the difference between the effect sizes of two studies.
2. Believing that if there is an effect in a situation, each study of that situation will show a significant effect – the chance to find an effect when there really is one is often quite low.

Significance testing: meta-analysis tries to record the actual level of significance obtained (instead of whether a study reached a certain level) on the standard normal deviate that corresponds to a p value. The use of signed normal deviates:

• Increase a study’s informational value.
• Increases a study’s utility.
• Changes a study’s cost-utility ratio and ethical value.

Meta-analyses increase research utility and ethical justification, providing accurate effect size estimates.

What are issues in reporting psychological research?

Misrepresenting findings

Some misrepresentations of findings are more obviously unethical than others.

• Intentional misrepresentation: includes fabricating data, intentionally/knowingly allocating subjects to experimental and control conditions to support the hypothesis. Not being blind to treatment condition and recording responses, and research assistants recording responses knowing the hypothesis and treatment conditions.
• Unintentional misrepresentation: recording, computational, and data analytic errors can all lead to inaccurate results. This includes causist language and questionable generalizability. Errors in data diminish research utility and shift cost-utility ratio in an unfavourable direction.

Misrepresenting credit

• Problems of authorship: many papers are multi-authored making it difficult to allocate authorship credit. Who becomes a coauthor vs a footnote? Who is assigned first or last coauthor in the listing?
• Problems of priority: an issue between research groups. Who got the idea first?

Failing to report/publish

What was not reported and why? The two biggest forms of failure to report are self-censoring and external censoring.

• Self-censoring: can be admirable when a study is done badly, could be a service to science to start over. Less admirable reasons could be failing to report data that contradict earlier research or personal values – poor science and ethics.
  • Good practice is to report all results that give information on the hypothesis and give data that other researchers could use.
• External censoring: progress and the slowing of progress in science depend on external censoring. Sciences would maybe be more chaotic if it weren’t for censorship by peers who keep bad research from being released. Two major bases for external censorship:
  • Evaluation of methodology used in a study.
  • Evaluation of results obtained in a study.

What can we conclude?

The ethical quality of our research is not independent of the scientific quality of our research.

What is the problem?

The experimental method tries to abstract certain variables from complicated situations in nature, and reproduce parts of them in the lab in order to determine the effect of certain variables. This allows for generalization from information obtained in the lab back to the original situation.

Subjects being observed are usually viewed as passive responders instead of active participants. Additionally, experimental stimuli tend to be defined in terms of what is done to the subject, not how they react (limited dynamism). These factors combined lead to issues in reproducibility and ecological validity, which are necessary for meaningful experimentation.

What are demand characteristics?

A participant actively wants to contribute to research and will react to ‘demand characteristics’. A participant’s behaviour in an experiment is determined by perceived demand characteristics of the setting as well as by experimental variables.

Demand characteristics are the total cues conveying a hypothesis to participants which determine their behaviour (rumours about experiment, explicit/implicit information given during the experiment, laboratory setting, views based on previous knowledge and experience of participant etc.). the perceived demand characteristics in any experiment can vary.

What factors affect a subject’s reaction to stimuli in an experimental setting?

• Motivation to comply with experimenter’s instructions.
• Perception of behaviour research.
• Nature of cues likely picked up etc.

It is proposed that these factors must be further elaborated and the parameters of an experimental setting should be more carefully defined so that sufficient controls can be designed to isolate the effects of the setting from the effects of the experimental variables.

In our culture the roles of subject and experimenter are well understood and carry well-defined mutual role expectations within an experimental setting:

• Subjects put themselves under experimenter’s control.
• Subjects tolerate boredom/discomfort if required.
• Shared assumption of the purpose justifies request for many things from a participant in an experiment.
• Effort and discomfort are justified by a higher purpose.

It is hard to design an experiment that tests the degree of social control in an experiment because of the already high degree of control in an experimental situation.

Why do people comply with such a wide range of instructions/requests? (motivational aspects)

• They want the experiment to succeed and be a ‘useful’ participant.
• They attribute meaning to their actions.
• They identify with scientific goals.
• Subjects are concerned with their performance: reinforcing self-image, concerned with the usefulness of their performance.
• Subjects will act in line with ‘being a good participant’ and in a way to validate the perceived/assumed hypothesis.

In what kind of contexts and circumstances do demand characteristics become significant in determining the behaviour of subjects in experimental situations?

• Demand characteristics can not be eliminated, but you can study their effect.
• Studies need to take the effects of demand characteristics into account.
• Response to demand characteristics is not just conscious compliance – perception of experimenter’s hypothesis tends to be a more accurate predictor of actual performance than a statement about what the participants think they have done on a task.
• The most powerful demand characteristics, in determining behaviour, convey the experiment’s purpose effectively, but not obviously. If the subject knows the experimenter’s expectations, there may be a tendency for biasing in the opposite direction.

What are experimental techniques to study demand characteristics?

1. Studying each participant’s perception of the experimental hypothesis. Determine how they correlate with observed behaviour.
2. Determine which demand characteristics are perceived, post-experimental inquiry – inquiry procedures required to not provide expectation cues.
   • Criticisms of this inquiry: (a) inquiry procedure is subject to demand characteristics and (b) participant’s won behaviour can influence their perception of the purpose of the study. So a correlation of the participant’s expectations vs. experimenter’s hypothesis may not have much to do with determining behaviour.
3. Pre-experimental inquiry (approximation, not proof) – show procedure, materials, requirements, then ask participant’s what the hypothesis is without them having done experimental tasks. Also ask how they would have asked it if they had been part of the experiment.
   • Leads to formulating possible hypotheses without knowing about their own reactions to testing.
   • Participant’s own behaviour in the experiment can’t influence their perception of the purpose of the study because they did not take part in it yet.
   • Participants of pre-experimental inquiry can’t take part in the actual experiment anymore, but must be drawn from the same population.
   • If subjects describe behaviour matching the one actually performed at testing, it becomes possible that demand characteristics are responsible for behaviour. It ispossible that demand characteristics, instead of experimental variables, could account for the behaviour of an experimental group.
4. Hold the demand characteristics constant and get rid of the experimental variable. Simulating participants: instructed to act as if exposed to experimental variable.
5. Blind design controls for experimenter bias is important, as experimenter bias will also influence the participant’s assumed hypothesis.

All these techniques depend on active cooperation of the control participants. This suggests that the subject does not just respond in these control situations but rather is required to actively solve a problem. We should view experiments as a special form of social interaction. It is suggested that control of demand characteristics could lead to greater replicability and generalizability of psychological experiments.

Summary and outlook

• Experiments as a special form of social interaction.
• Subject should be recognized as an active participant.
• Subject’s behaviour in an experiment is a function of the totality of the situation, including the experimental variables and demand characteristics.
• Control for demand characteristics is suggested to lead to higher replicability and ecological validity.
• Study and control of demand characteristics are not just a matter of good research technique; it is an empirical issue to figure out in what context they significantly influence participants’ experimental behaviour.

What is the purpose of this paper?

A study with low power has less likelihood of detecting a true effect, but it is less well appreciated that low power also decreases the likelihood that a significant result reflects a true effect.  This paper shows that average statistical power of studies in neurosciences is low. Consequences include: overestimates of ES and low reproducibility. There are also ethical dimensions: unreliable research and inefficient/wasteful.

The paper discusses issues that arise when low-powered research is frequent. They are divided into two categories: (1) concerning problems that are mathematically expected even if the research is otherwise perfect, (2) concerning problems that reflect biases tending to co-occur with studies of low power or become worse in small, underpowered studies.

What are the three main problems?

The main problems contributing to producing unreliable findings with low power are:

• Low probability of finding true effects
• Low positive predictive value (PPV) when an effect is claimed
• An exaggerated estimate of the effect size when a true effect is discovered

What are some key statistical terms?

• CAMARADES: collaborative approach to meta-analysis and review of animal data from experimental studies. A collaboration aiming to decrease bias and improve method quality and reporting in animal research. Promotes data-sharing.
• Effect size: standardized measure quantifying the size of a difference between two groups or strength of an association between two variables.
• Excess significance: phenomenon where literature has an excess of statistically significant results due to biases in reporting. Mechanisms contributing to bias: study publication bias, selective outcome reporting, and selective analysis bias.
• Fixed and random effects: a fixed-effect meta-analysis assumes that the underlying effect is the same in all studies and that any variation is because of sampling errors.
• Meta-analysis: statistical methods for contrasting and combining results from different studies to give more powerful estimates of the true ES.
• Positive predictive value (PPV): probability that a ‘positive’ finding reflects a true positive effect; depends on prior probability of it being true.
• Proteus phenomenon: the situation where the first published study is often most biased towards an extreme result (winner’s curse). Replication studies tend to be less biased toward the extreme – find smaller ES’s of contradicting effects.
• Statistical power: probability that a test will correctly reject a false H0. The probability of not making a type II error (probability making type II error = β and power = 1 – β).
• Winner’s curse: the ‘lucky’ scientists who makes a discovery is cursed by finding an inflated estimate of the effect. Most severe when the thresholds (significance) are strict and studies too small – low power.

What are additional biases associated with low power?

Firstly, low-powered studies are more likely to give a wide range of estimates of the magnitude of an effect. This is known as ‘vibration of effects’: situation where a study obtains different estimates of an effect size depending on the analytical options it implements. Results can vary depending on the analysis strategy, especially in small studies.

Secondly, publication bias, selective data analysis, and selective reporting of outcomes are likely to affect low-powered studies.

Third, small studies may be lower quality in other aspects of their design as well. These factors can further exacerbate the low reliability of evidence attained in studies with low power.

What are the implications for the likelihood that a research finding reflects a true effect?

Trying to establish the average power in neuroscience is hampered by the issue that the ES is unknown. One solution may be using data from meta-analysis (estimate ES). Studies contributing to meta-analysis, however, are subject to the same problem of unknown ES.

Results show that the average power of studies in neuroscience is probably not more than between 8%-31%. This has major implications for the field.

• Likelihood that any significant finding is actually a true effect is small (PPV decreases when power decreases).
• Ethical implications: inefficient and wasteful in animal/human testing.

What recommendations are there for researchers?

• Perform an a priori power calculation: estimate ES you are looking for and design your study accordingly.
• Disclose methods and findings transparently.
• Pre-register your study and analysis plan: this clarifies confirmatory and exploratory analyses, reducing opportunities for non-transparency.
• Make study materials and data available – improving quality of replication.
• Work collaboratively to increase power and replicate findings; combining data increases total N (and power) while minimizing labour/resources.

What can we conclude?

Researchers can improve confidence in published reports by noting in the test how they determined their sample size, all data exclusions, all data manipulations, and all measures in the study. When stating that is not possible, disclosure of rationale and justification of deviations will improve readers’ understanding and interpretation of reported effects and what level of confidence in the reported effects is appropriate.

What is the purpose of the article?

The article discusses causal inference based on observational data, introducing readers to graphical causal models that can provide a powerful tool for thinking more clearly about the interrelations between variables.

Researchers from different areas use different strategies to deal with weak observational data (manipulating the independent variable can sometimes be unfeasible, unethical, or impossible).

• E.g. using surrogates can lead to valuable insights but also comes with a trade-off of decreased external validity.
• Some researchers try to cautiously avoid causal language.
• Many have tried to statistically control for third variables. Often these attempts lack proper justification.

This article aims to give psychologists a primer to a more principles approach to make causal inferences based on observational data. It discusses the improvement of causal inferences on observational data through using directed acyclic graphs (DAGs). They provide visual representations of causal assumptions.

What are directed acyclic graphs?

DAG’s consist of nodes (representing variables in research) and arrows (indicating the direction of the relationship). It can display the direction of causation or a lurking (confounding) variable and only contain one headed arrow. They are ‘acyclic’ because they don’t allow for cyclic paths in which variables become their own ancestors. A variable cannot causally affect itself.

What is a back-door path?

If we want to derive a valid causal conclusion we need a causal DAG that’s complete because it includes all common causes of all pairs of variables that are included in the DAG. After such a DAG is built, ‘back-door paths’ can be recognized. These are (non-causal) paths starting with an arrow pointing to the independent variable and ending with an arrow pointing to the dependent variable (indicating there may be a common factor affecting both treatment and outcome). They are problematic whenever they convey an association and can show a spurious association.

The purpose of third variable control is to block back-door paths. If all back-door paths between variables can be blocked, the causal effect connecting the independent and dependent variables can be identified.

How do we control for a variable?

• Stratified analysis: stratify the sample controlling for confounders. Maybe unfeasible if the third variable needing control has many levels, if it is continuous, or if multiple third variables and interactions need to be accounted for.
• Including third variables in regression models: dependent variable can be regressed on both the independent variable and the covariate to control for the covariate’s effects. Does not guarantee adequate adjustment for the covariate.
• Matching: when there is a need to control for many third variables. Propensity-score matching is popular in social sciences but it fails to properly identify the causal effect.

What are examples of collider bias?

• Nonresponse bias: example, if a researcher analyzes only completed questionnaires, and the variables of interest are associated with questionnaire completion. Assuming we are interested in the association between grit and intelligence and our assessment is burdensome. Grit and intelligence make it easier for respondents to push through and finish it. Questionnaire completion is therefore a collider between grit and intelligence.
• Attrition bias: systematic errors caused by unequal loss of participants. If only remaining respondents are included in analysis, spurious associations may arise and open up back-door paths between variables of interest.

What are some definitions?

• Ancestor: variable causally affecting another variable, influencing it directly (ancestor > X) or indirectly (ancestor > mediator > X). direct ancestors are called parents.
• Blocked path: path containing (a) collider that the analysis has not been conditioned on or (b) a non-collider (confounder or mediator) that the analysis has been conditioned on. Does not transmit an association between variables.
• Causal path: path consisting of only chains that can convey a causal association if unblocked.
• Chain: causal structure of the form A > B > C.
• Collider: a variable in the middle of an inverted fork (A > collider
• Conditioning on a variable: process of introducing information about a variable into an analysis (statistical control or sample selection).
• Confounder: variable in the middle of a fork (A C).
• Descendant: variable causally affected by another variable, directly (X > descendant) or indirectly (X > mediator > descendant). Direct descendants are called children.
• Fork: structure of the form A C.
• Inverted fork: structure of the form A > B
• Mediator: variable middle in a chain (A > mediator > C.
• Node: variable in a DAG.
• Non-causal path: path containing at least one fork or inverted fork and can show a non-causal association when unblocked.

What are some examples of DAGs?

• Educational attainment grades (fork).
• Intelligence > educational attainment > grades (chain).
• Intelligence > grades

Summary

The practice of making causal inferences based on observational data depends on awareness of potential confounders and meaningful statistical control (or non-control) taking into account estimation issues like nonlinear confounding and measurement error. Back-door paths should be considered before data is collected to make sure all relevant variables are measured. Additionally, variables that should not be controlled for (colliders and mediators) need to be considered.