Funder et al. (2014). Improving the dependability of research in personality and social psychology: Recommendations for research and educational practice.” - Article summary

There appears to be a high rate of false positives in science. The p-value refers to the conditional probability that the present data or more extreme will be observed in a random sample given that the null hypothesis is true. Critique of using the p-value includes not interpreting the p-value properly (1) and the fact that the p-value varies with a varying sample size (2).

The effect size refers to the magnitude of the found statistical relationship. Unstandardized effect sizes are preferred when there is a clear consensus regarding that the measurement unit is in the interval level (e.g. seconds; blood pressure).When this is not the case (e.g. in psychology), standardized effect sizes should be used. The effect size can be easily interpreted if the sample sizes are equal and the total sample size is moderate to large. However, this becomes more complex if the sample sizes differ greatly.

Power refers to the probability that a true effect of a precisely specified size in the population will be detected using significance testing. It is the probability of finding an effect given that an effect of the specified size (i.e. the power) exists. The statistical power is one minus the type II error rate. The type II error rate refers to the probability that a true effect will not be detected using significance testing. It is the probability that the alternative hypothesis is wrongfully rejected.

Power should be maximized in a study. The power, however, is affected by the sample size (1), the measurement error (2) and the homogeneity of the participants (3). According to Cohen, research should use power of at least 0.8.

It is problematic to solely focus on the observed p-value because findings with equivalent p-values can have very different implications. In cases with large sample sizes, very small effect sizes can be significant and in cases with small sample sizes, a significant effect may produce an implausible large effect size. Therefore, there is a big difference between practical and statistical significance. A lot of research with small sample sizes is underpowered.

There are several recommendations to improve research:

  1. Describe and address choice of N and consequent issues of statistical power
    The researchers should design studies with sufficient power to detect the key effects of interest. The sample size should be justified on the smallest effect of interest. More studies with adequate power means more statistical effects of interest achieve significance.
  2. Report effect sizes and 95% CIs for reported findings
    The p-values should be supplemented by effect sizes that provide information on the magnitude of the finding.
  3. Avoid questionable research practices
    Research practices that tweak the results afterwards and undermine the researcher’s ability to draw a valid conclusion should be avoided. Tweaked data leads to a larger effect size than the true effect size.
  4. Include in an appendix the verbatim wording of all independent and dependent variable instructions, manipulations and measures
    This allows other researchers to replicate the study as the precise procedures are made known to the public.
  5. Use open access data
    All data should be shared if the anonymity of the participants can be protected (1), it is not illegal (2) and the data requesting researcher agrees with the previous two points (3). This makes it easier to replicate studies and confirm findings.
  6. Encourage and improve the availability of publication outlets for replication studies
    Replicating more studies leads to a better foundation on which to base novel research.
  7. Maintain flexibility and openness to alternative standards and methods when evaluating research
    The rules should not be rigid and should be open to change, if a proper argument is presented.

Conducting multiple tests of significance on a dataset without statistical correction (1), running participants until significant results are obtained (2), dropping observations, measures, items, conditions or participants after looking at the effects on the outcomes of interest (3) and running multiple experiments with similar procedures and only reporting those with significant results (4) are questionable research practices.

There are several recommendations for educational practice:

  1. Encourage a culture of ‘getting it right’ rather than finding significant results
    Accurate knowledge should be supported over significant results. The contribution of a particular piece of research should be evaluated in terms of whether the research is carefully designed to address important questions and whether it makes use of an appropriate sample.
  2. Teach and encourage transparency of data reporting, including imperfect results
    There should be a focus on informativeness of data. All the information should be available and not only information that helps the flow of the paper or the career of the researcher.
  3. Improve methodological instructions on topics such as effect size, confidence intervals, statistical power, meta-analysis, replication and the effects of questionable research practices
    There should be a training regarding the issues in psychology described in the recommendations in the (under)graduate courses of students.
  4. Model sound science and support junior researchers who seek to get it right
    The current research practice should be modelled after the improved version and the new researchers who seek to get it right should be supported.

Replication studies might need to use a lower than reported effect size (1), develop a minimum value for an effect size that is deemed to small (2) and use a higher value than 0.80 for statistical power (3). This makes sure that replication is possible because if this is not done, the probability of finding a significant result decreases steadily. These steps enhance the credibility and the usefulness of a replication study.

 

Image

Access: 
Public

Image

Join WorldSupporter!
This content is used in:

Evidence-based Clinical Practice – Full course summary (UNIVERSITY OF AMSTERDAM)

Evidence-based Clinical Practice – Article overview (UNIVERSITY OF AMSTERDAM)

Search a summary

Image

 

 

Contributions: posts

Help other WorldSupporters with additions, improvements and tips

Add new contribution

CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Image CAPTCHA
Enter the characters shown in the image.

Image

Spotlight: topics

Check the related and most recent topics and summaries:
Institutions, jobs and organizations:
Activity abroad, study field of working area:
This content is also used in .....

Image

Check how to use summaries on WorldSupporter.org

Online access to all summaries, study notes en practice exams

How and why use WorldSupporter.org for your summaries and study assistance?

  • For free use of many of the summaries and study aids provided or collected by your fellow students.
  • For free use of many of the lecture and study group notes, exam questions and practice questions.
  • For use of all exclusive summaries and study assistance for those who are member with JoHo WorldSupporter with online access
  • For compiling your own materials and contributions with relevant study help
  • For sharing and finding relevant and interesting summaries, documents, notes, blogs, tips, videos, discussions, activities, recipes, side jobs and more.

Using and finding summaries, notes and practice exams on JoHo WorldSupporter

There are several ways to navigate the large amount of summaries, study notes en practice exams on JoHo WorldSupporter.

  1. Use the summaries home pages for your study or field of study
  2. Use the check and search pages for summaries and study aids by field of study, subject or faculty
  3. Use and follow your (study) organization
    • by using your own student organization as a starting point, and continuing to follow it, easily discover which study materials are relevant to you
    • this option is only available through partner organizations
  4. Check or follow authors or other WorldSupporters
  5. Use the menu above each page to go to the main theme pages for summaries
    • Theme pages can be found for international studies as well as Dutch studies

Do you want to share your summaries with JoHo WorldSupporter and its visitors?

Quicklinks to fields of study for summaries and study assistance

Main summaries home pages:

Main study fields:

Main study fields NL:

Follow the author: JesperN
Work for WorldSupporter

Image

JoHo can really use your help!  Check out the various student jobs here that match your studies, improve your competencies, strengthen your CV and contribute to a more tolerant world

Working for JoHo as a student in Leyden

Parttime werken voor JoHo

Statistics
1992