Researchers should not compare artificially created groups in an experiment (e.g. based on the median). There are several problems with median-splits:

1. Median splits change the original information drastically
2. Effect sizes get smaller
3. There is an increased chance of finding spurious effects

CATEGORICAL PREDICTORS IN THE LINEAR MODEL
Comparing the difference between the means of two groups is predicting an outcome based on membership of two groups. A t-statistic is used to ascertain whether a model parameter is equal to zero. In other words, the t-statistic tests whether the difference between group means is equal to zero.

THE T-TEST
There are two types of t-tests:

1. Independent t-test (independent measures t-test)
   This is comparing two means in which each group has its own set of participants.
2. Paired-samples t-test (dependent t-test)
   This is comparing two means in which each group uses the same participants.

The t-test is used to see whether there is an actual difference between two groups (e.g. experimental and control). If there is no difference between the two groups, we expect to see the same mean. There is natural variation in each sample, so the mean is (almost) never exactly the same. Therefore, just by looking at the means, it is impossible to state whether there is a significant difference between two groups. In the t-test, a set level of confidence (normally 0.95), alpha, is used as a threshold of when the difference is significant. The t-statistic is used to compute a p-value and this p-value is compared to the alpha. If the p-value is equal to or smaller than the alpha, it means that there is a significant difference between the two means and then we state that there is an actual difference. The larger the difference between two means relative to the standard error, the more likely it is that there is an actual difference between the two means.

The t-test is always computed under the assumption that the null hypothesis is true. It uses the following general formula:

The null hypothesis usually states that there is no difference between the two means, meaning that the null hypothesis mean would equal ‘0’. The standard error of the sampling distribution is the standard error of differences. The standard error helps the t-test because it gives a scale of likely variability between samples.

The variance sum law states that the variance of a difference between two independent variables is equal to the sum of their variances (e.g. the variance of x1-x2 = variance of x1 + variance x2). The variance of the sampling distribution of difference between two sample means is equal to the sum of variances of the two populations from which the samples were taken. This leads to the following formula for the standard error:

This equation holds if the sample sizes are equal. If this is not the case, a pooled variance estimate is used instead.

This pooled variance is then used in the formula for the t-statistic:

PROCEDURE OF THE T-TEST
The paired-samples t-test uses the assumption of normality to the sampling distribution of the differences between the scores. The procedure of the t-test is the following. The data is explored (1), the t-test is conducted (2) and the effect size is calculated (3).

Levene’s test tests whether variances are different in different groups.

ROBUST AND BAYESIAN TESTS OF TWO MEANS
If a confidence interval contains ‘0’ it is possible that the two groups are the same as there is a possibility of the difference between the groups being ‘0’. The Bayes’ factor (e.g. 0.419) mean that the data are 0.419 times as probable under the null hypothesis as under the alternative.

The t-value can be transformed into an r-value using the following formula:

The r-value represents effect-size. When computing Bayesian factors, they have to be interpreted in the following way. A Bayesian factor can be 0.005. This means that the data are 0.005 times as probable under the null hypothesis as under the alternative hypothesis. If you turn this around (by dividing 1/0.005), then you get 200. This means that the data are 200 times as probable under the alternative hypothesis as under the null hypothesis. This provides strong evidence in favour of the alternative hypothesis.

Using a t from a paired-samples t-test leads to an overestimation of the population effect size. This does not occur when using an independent sample t-test. It could be useful to use Cohen’s d to estimate the effect size when computing a paired samples t-test.