Understanding effect size, proportion of explained variance and power of tests to your significant results

Effect size, proportion of explained variance and power of tests

Effect size (Cohen's d)

Some researchers critize the process of testing hypothesis. The main critique refers to the interpretation of a significant result. When testing hypotheses, most attention is paid to the data instead of to the hypotheses. When the null hypotheses is rejected, we make statements about the sample data and not about the null hypothesis. Based on the sample data, the null hypothesis is rejected or not. We do not know whether the null hypothesis is truly false or true. Another point of critique is that a significant effect does not imply anything about the effect size. Something is significant or not, but it does not imply anything about the size of the effect. Thus, a significant effect is not equal to a large effect. To provide more insight into the size of an effect, Cohen (1988) proposed the so-called effect size. His measure for effect size is called Cohen’s d.

\[Cohen's\: d =\frac{(\bar{x} - \mu)}{\sigma}\]

  • Cohen's d: standardised difference between two means
  • : sample mean
  • µ: population mean
  • σ: population standard deviation

The outcome of Cohen’s d is classified as a small effect for d = 0.2, a medium effect for d = 0.5 and a large effect for d = 0.8.

Please note: APA style strongly recommends use of η2 instead of Cohen's d.

Proportion of explained variance (r2)

A different way to determine the effect size, is by looking at how much variance between the scores is explained by the effect. The proportion of explained variance can be found by squaring the t-statistic and dividing it by the same number plus the degrees of freedom. In formula:

\[r^2 = \frac{t^2}{t^2 + df}\]

  • r2: proportion of explained variance
  • t: t-statistic
  • df: degrees of freedom: n-1

A proportion explained variance of 0.01 refers to a small effect. A value of 0.09 refers to a medium effect. A proportion of 0.25 refers to a large effect. The r2 is usually presented in percentages in the literature.

Confidence intervals

Confidence intervals can assist in describing the results from hypothesis tests. When we obtain a specific estimation of a parameter, we call this a point estimation. Next, there are interval estimations, which obtain the limits within the true population parameter(µ) likely is. These are called the confidence limits, that make the confidence interval. We want to know how high and how low the µ-value can be, for which we do not reject H0. This provides the limits within we keep the null hypothesis.

Power

Besides measuring the effect size, it is also possible to measure the power of a statistical test. Power refers to the extent to which a study is capable of detecting the effects in the examined variable. A study with a high power is able to detect existing effects, while a study with a low power will likely not detect these effect. The power is influenced by many things. One of these is the number of participants. In general, it applies that the more participants there are, the higher the power is. Strong effects are easier to identify than weak effects. A study with a low power will often not identify weak effects, but may identify the strong effects. To identify a weak effect, a high power is required. For identifying weak effects, it is also useful to have data from many participants. Power can be calculated as:

\[Power = 1 - \beta\]

  • β: the chance of a type-II error

Researcher often require a power of 0.80. The power of the test is influenced by three factors:

  1. First, the sample size is important. The larger the sample size, the higher the chance of rejecting the null hypothesis when the null hypothesis is actually false. This means that the power of the test increases when the sample size increases.

  2. Second, the power of the test decreases when the alfa level is lowered. When the alfa for example is decreased from 5 to 1%, the chance that a true effect is found (thus that the null hypothesis is rejected correctly) decreases.

  3. Third, the power increases when a two-sided test is transferred to a one-sided test.

Statistics: suggestions, summaries and tips for encountering Statistics

Statistics: suggestions, summaries and tips for encountering Statistics

Knowledge and assistance for discovering, identifying, recognizing, observing and defining statistics.

Startmagazine: Introduction to Statistics
Stats for students: Simple steps for passing your statistics courses

Stats for students: Simple steps for passing your statistics courses

Image

How to triumph over the theory of statistics (without understanding everything)?

Stats of students

  • The first years that you follow statistics, it is often a case of taking knowledge for granted and simply trying to pass the courses. Don't worry if you don't understand everything right away: in later years it will fall into place, and you will see the importance of the theory you had to know before.
  • The book you need to study may be difficult to understand at first. Be patient: later in your studies, the effort you put in now will pay off.
  • Be a Gestalt Scientist! In other words, recognize that the whole of statistics is greater than the sum of its parts. It is very easy to get hung up on nit-picking details and fail to see the forest because of the trees
  • Tip: Precise use of language is important in research. Try to reproduce the theory verbatim (i.e. learn by heart) where possible. With that, you don't have to understand it yet, you show that you've been working on it, you can't go wrong by using the wrong word and you practice for later reporting of research.
  • Tip: Keep study material, handouts, sheets, and other publications from your teacher for future reference.

How to score points with formulas of statistics (without learning them all)?

  • The direct relationship between data and results consists of mathematical formulas. These follow their own logic, are written in their own language, and can therefore be complex to comprehend.
  • If you don't understand the math behind statistics, you don't understand statistics. This does not have to be a problem, because statistics is an applied science from which you can also get excellent results without understanding. None of your teachers will understand all the statistical formulas.
  • Please note: you will probably have to know and understand a number of formulas, so that you can demonstrate that you know the principle of how statistics work. Which formulas you need to know differs from subject to subject and lecturer to lecturer, but in general these are relatively simple formulas that occur frequently, and your lecturer will likely tell you (often several times) that you should know this formula.
  • Tip: if you want to recognize statistical symbols, you can use: Recognizing commonly used statistical symbols
  • Tip: have fun with LaTeX! LaTeX code gives us a simple way to write out mathematical formulas and make them look professional. Play with LaTeX. With that, you can include used formulas in your own papers and you learn to understand how a formula is built up – which greatly benefits your understanding and remembering that formula. See also (in Dutch): How to create formulas like a pro on JoHo WorldSupporter?
  • Tip: Are you interested in a career in sciences or programming? Then take your formulas seriously and go through them again after your course.

How to practice your statistics (with minimal effort)?

How to select your data?

  • Your teacher will regularly use a dataset for lessons during the first years of your studying. It is instructive (and can be a lot of fun) to set up your own research for once with real data that is also used by other researchers.
  • Tip: scientific articles often indicate which datasets have been used for the research. There is a good chance that those datasets are valid. Sometimes there are also studies that determine which datasets are more valid for the topic you want to study than others. Make use of datasets other researchers point out.
  • Tip: Do you want an interesting research result? You can use the same method and question, but use an alternative dataset, and/or alternative variables, and/or alternative location, and/or alternative time span. This allows you to validate or falsify the results of earlier research.
  • Tip: for datasets you can look at Discovering datasets for statistical research

How to operationalize clearly and smartly?

  • For the operationalization, it is usually sufficient to indicate the following three things:
    • What is the concept you want to study?
    • Which variable does that concept represent?
    • Which indicators do you select for those variables?
  • It is smart to argue that a variable is valid, or why you choose that indicator.
  • For example, if you want to know whether someone is currently a father or mother (concept), you can search the variables for how many children the respondent has (variable) and then select on the indicators greater than 0, or is not 0 (indicators). Where possible, use the terms 'concept', 'variable', 'indicator' and 'valid' in your communication. For example, as follows: “The variable [variable name] is a valid measure of the concept [concept name] (if applicable: source). The value [description of the value] is an indicator of [what you want to measure].” (ie.: The variable "Number of children" is a valid measure of the concept of parenthood. A value greater than 0 is an indicator of whether someone is currently a father or mother.)

How to run analyses and draw your conclusions?

  • The choice of your analyses depends, among other things, on what your research goal is, which methods are often used in the existing literature, and practical issues and limitations.
  • The more you learn, the more independently you can choose research methods that suit your research goal. In the beginning, follow the lecturer – at the end of your studies you will have a toolbox with which you can vary in your research yourself.
  • Try to link up as much as possible with research methods that are used in the existing literature, because otherwise you could be comparing apples with oranges. Deviating can sometimes lead to interesting results, but discuss this with your teacher first.
  • For as long as you need, keep a step-by-step plan at hand on how you can best run your analysis and achieve results. For every analysis you run, there is a step-by-step explanation of how to perform it; if you do not find it in your study literature, it can often be found quickly on the internet.
  • Tip: Practice a lot with statistics, so that you can show results quickly. You cannot learn statistics by just reading about it.
  • Tip: The measurement level of the variables you use (ratio, interval, ordinal, nominal) largely determines the research method you can use. Show your audience that you recognize this.
  • Tip: conclusions from statistical analyses will never be certain, but at the most likely. There is usually a standard formulation for each research method with which you can express the conclusions from that analysis and at the same time indicate that it is not certain. Use that standard wording when communicating about results from your analysis.
  • Tip: see explanation for various analyses: Introduction to statistics
Statistics: suggestions, summaries and tips for understanding statistics

Statistics: suggestions, summaries and tips for understanding statistics

Knowledge and assistance for classifying, illustrating, interpreting, demonstrating and discussing statistics.

Startmagazine: Introduction to Statistics
Understanding data: distributions, connections and gatherings
Understanding reliability and validity
Statistics Magazine: Understanding statistical samples
Understanding distributions in statistics
Understanding variability, variance and standard deviation
Understanding inferential statistics
Understanding type-I and type-II errors
Understanding effect size, proportion of explained variance and power of tests to your significant results
Statistiek en onderzoek - Thema
Statistics: suggestions, summaries and tips for applying statistics

Statistics: suggestions, summaries and tips for applying statistics

Knowledge and assistance for choosing, modeling, organizing, planning and utilizing statistics.

Applying z-tests and t-tests
Applying correlation, regression and linear regression
Applying spearman's correlation - Theme
Applying multiple regression

More knowledge and assistance for Encountering, Understanding and Applying Statistics

Realising powerful statistics

What can you do on a WorldSupporter Statistics Topic?