Comparing two means - summary of chapter 10 of Statistics by A. Field (5th edition)

Statistics
Chapter 10
Comparing two means

Categorical predictors in the linear model

If we want to compare differences between the means of two groups, all we are doing is predicting an outcome based on membership of two groups.
This is a linear model with one dichotomous predictor.

The t-test

Independent t-test: used when you want to compare two means that come from conditions consisting of different entities (this is sometimes called the independent-measures or independent-means t-test)
Paired-samples t-test: also known as the dependent t-test. Is used when you want to compare two means that come from conditions consisting of the same or related entities.

Rationale for the t-test

Both t-tests have a similar rationale:

• two samples of data are collected and the sample means calculated. These might differ by either a little or a lot
• If the samples come from the same population, then we expect their means to be roughly equal. Although it is possible for the means to differ because of sample variation, we would expect large differences between sample means to occur very infrequently. Under the null hypothesis we assume that the experimental manipulation has no effect on the participant’s behaviour: therefore, we expect means from two random samples to be very similar.
• We compare the difference between the sample means that we collected to the difference between the sample means that we would expect to obtain (in the long run) if there were no effect. We use the standard error as a gauge of the variability between sample means. If the standard error is small, then we expect most samples to have very similar means. When the standard error is large, large differences in sample means are more likely. If the difference between the samples we have collected is larger than we would expect based on the standard error then one of two things has happened:
  • There is no effect but sample means form our population fluctuate a lot and we happen to have collected two samples that produce very different means.
  • The two samples come from different populations, which is why they have different means, and this difference is indicative of a genuine difference between the samples.
• the larger the observed difference between the sample means, the more likely it is that the second explanation is correct.

Most test statistics have a signal-to-noise ratio: the ‘variance explained by the model’ divided by the ‘variance that the model can’t explain’.
Effect divided by error.
When comparing two means, the model we fit is the difference between the two group means. Means vary from sample to sample (sampling variation) and we can use the standard error as a measure of how much means fluctuate. Therefore, we can use the standard error of the differences between the two means as an estimate of the error in our model.

T= (observed difference between sample means – expected difference between population means if null hypotheses is true) / estimate of the standard error of the difference between two samples

The top half of the equation is the ‘model’, which is that the difference between means is bigger than the expected value under the null hypothesis, which in most cases will be 0.
The bottom half is the ‘error’.
The exact form that this equation takes depends on whether scores are independent or related to each other.

The paired-samples t-test equation explained

On average, sample means will be very similar to the population mean. Therefore, on average, most samples should have very similar means.
Our pair of random samples should have similar mean, meaning that the difference between means is zero, or close to zero.

Bottom half
If we plotted the frequency distribution of differences between means of pairs of samples we’d get the sampling distribution of differences between means. Most around zero.
Standard error of differences: the standard deviation of the sampling distribution.
The standard error helps us gauge by giving us a scale of likely variability between samples.

• if the standard error is small, then we know that even a modest difference between scores in the two conditions would be unlikely from two random samples
• if the standard error is large then a modest difference between scores is plausible from two random samples.

The standard error of differences provides a scale of measurement for how plausible it is that an observed difference between sample means could be the product of taking two random samples from the same population.

Top half
The size of the observed effect.
For each person, if we took their score in one condition and subtracted it from their score in the other, this would give us a difference score for each person.

T is the signal-to-noise ratio or the systematic variance compared to the unsystematic variance.
The top half is the signal or effect.
The bottom places that effect in the context of the natural variation between samples.

If the experimental manipulation creates difference between conditions, then we would expect the effect to be greater than the unsystematic variation and, at the very least, t will be greater than 1.

We can compare the obtained value of t against the maximum vale we would expect to get if the null hypothesis were true in a t-distribution with the same degrees of freedom.
If the observed t exceeds the critical value for the predetermined alpha, scientists tend to assume that this reflects an effect of their independent variable.

The independent t-test equation explained

When scores in two groups come form different participants, pairs of scores will differ not only because of the experimental manipulation reflected by those conditions, but also because of other sources of variance.
These individual differences are eliminated when we use the same participants across conditions.

We compare differences between two sample means and not between individual pairs of scores.
The differences between sample means is compared to the difference we would expect to get between the means of the two populations form which the samples come.

Having converted to variances, we can take advance of the variance sum law.
Variance sum law: states that the variance of a difference between two independent variables is equal to the sum of their variances.
The variance of the sampling distribution of differences between two sample means will be equal to the sum of variances of the two populations from which the samples were taken.
We can estimate the variance of the sampling distribution of differences by adding together the variances of the sampling distributions of the two populations.

We convert this variance back to a standard error by taking the square root.

This is only true when the sample sizes are equal, which in naturalistic studies may not be possible. To compare two groups that contain different numbers of participants we use a pooled variance estimate instead, which takes account of the difference in sample size by weighting the variance of each sample by a function of the size of sample on which it is based.

Assumptions of the t-test

For the paired-samples t-test the assumption of normality relates to the sampling distribution of the differences between scores, not the scores themselves.

SPSS

Independent t-test

• the independent t-test compares two means, when those means have come from different groups of entities
• you should probably ignot the collum called Levene’s test for equality of variance and always look at the row in the table called Equal variances not assumed
• look at the collumn called Sig. If the value is less than 0.05 then the means of the two groups are significantly different.
• look at the table called Bootstrap for independent samples test to get a robust confidence interval for the different between means
• look at the values of the means to see how the groups differ
• a robust version of the test can be computed using syntax.
• a Bayes factor can be computed that quantifies the ratio of how probable the data are under the alternative hypothesis compared to the null

Paired-samples t-test

• the paired-samples t-test compares two means, when those two means have come from the same entities.
• look at the column labelled Sig. If the value is less than 0.05 then the means of the two conditions are significantly different.
• look at the values of the means to tell you how the conditions differ.
• look at the table labelled Bootstrap for paired samples to get a robust confidence interval for the difference between means
• a robust version of the test can be computed using syntax
• a Bayes facto can be computed that quantifies the ratio of how probable the data are under the alternative hypothesis compared to the null
• calculate and report the effect size too