Many statistical models try to predict an outcome from one or more predictor variables. Statistics includes five things: Standard error (S), parameters (P), interval estimates (I), null hypothesis significance testing (N) and estimation (E), together making SPINE. Statistics often uses linear models, as this simplifies reality in an understandable way.

All statistical models are based on the same thing:

The data we observe can be predicted from the model we choose to fit plus some amount of error. The model will vary depending on the study design. The bigger a sample is, the more likely it is to reflect the whole population.

PARAMETER
A parameter is a value of the population. A statistic is a value of the sample. A parameter can be denoted by ‘b’. The outcome of a model uses the following formula:

In this formula, ‘b’ denotes the parameter and ‘X’ denotes the predictor variable. This formula calculates the outcome from two predictors. Degrees of freedom relates to the number of observations that are free to vary. One parameter is hold constant and the degrees of freedom must be one fewer than the number of scores used to calculate that parameter because the last number is not free to vary.

ESTIMATION
The method of least squares is minimizing the sum of squared errors. The smaller the error, the better your estimate is. When estimating a parameter, we try to minimize the error in order to have a better estimate.

STANDARD ERROR
The standard deviation tells us how well the mean represents the sample data. The difference in means across samples is called the sampling variation. Samples vary because they include different members of a population. A sampling distribution is the frequency distribution of sample means from the same population. The mean of the sampling distribution is equal to the population mean. A standard deviation of the sampling distribution tells us how the data is spread around the population mean, meaning that the standard deviation of the sampling distribution is approximately equal to the standard deviation of the population. The standard error of the mean (SE) can be calculated by taking the difference between each sample mean and the overall mean, squaring these differences, adding them up and dividing them by the number of samples and taking the square root of it. It uses the following formula:

The central limit theorem states that when samples get large (>30), the sampling distribution has a normal distribution with a mean equal to the population mean and the following standard deviation:

If the sample is small (<30), the sampling distribution has a t-distribution shape.

INTERVAL ESTIMATES
It is not possible to know the parameters, thus confidence intervals are used. Confidence intervals

A good graph has the following properties:

1. Shows the data
2. Induce the reader to think about the presented data
3. Avoid distorting data
4. Present many numbers with minimum ink
5. Make large data sets coherent
6. Encourage the reader to compare different pieces of data
7. Reveal the underlying message of data

There are also some graph building guidelines:

1. If plotting two variables, never use 3-D plots
2. Do not use unnecessary patterns in the bars
3. Do not use cylinder shaped bars if that is not functional
4. Properly label the x- and y-axis.

HISTOGRAMS
There are different types of histograms:

1. Simple histogram
   Visualize frequencies of scores for a single variable
2. Stacked histogram
   Compare relative frequencies of scores across groups
3. Frequency polygon
   The same as a simple histogram, but uses a line, instead of a bar
4. Population pyramid
   Comparing distributions across groups and the relative frequencies of scores in two populations.

BOXPLOTS
A box-plot or box-whisker diagram  uses the median as the centre of the plot. It is surrounded by the quartiles which show 25% and 75% of the data. There are several types of boxplots:

1. 1-D boxplot
   A single boxplot for all scores of the chosen outcome
2. Simple boxplot
   Multiple boxplots for the chosen outcome by splitting the data by a categorical variable
3. Clustered boxplot
   A simple boxplot, but it splits the data by a second categorical variable.

BAR CHARTS
Bar charts are often used to display means. There are different types of bars:

1. Simple bar
   The means of scores across different groups or categories.
2. Clustered bar
   Different coloured bars to represent levels of a second grouping variable (e.g: film rating and excitement and enjoyment)
3. Stacked bar
   Clustered bar, but the bars are stacked.
4. Simple 3-D bar
   Second grouping variable is represented by an additional axis
5. Clustered 3-D bar
   A clustered bar, but an extra categorical variable can be added on an extra axis
6. Stacked 3-D bar
   A 3-D clustered bar, but the bars are stacked
7. Simple error bar
   A simple bar chart, but there is no bar, but a line and a dot
8. Clustered error bar
   Clustered bar chart, but a dot with an error band around it.

LINE CHARTS
Line charts are bar charts but with lines instead of bars. There are two types of line charts:

1. Simple line
   The means of scores across different groups of cases
2. Multiple line
   This is equivalent to the clustered bar chart.

SCATTERPLOTS
A scatterplot is a graph that plots each person’s score on one variable against their score on another. There are several types of scatterplots:

1. Simple scatter
   A scatterplot of

Non-parametric tests can be used when the assumptions of the regular statistical tests have been violated. Non-parametric tests use fewer assumptions and are robust. A non-parametric test has less power than parametric tests if the sampling distribution is normally distributed.

Ranking the data refers to giving the lowest score the rank of 1, the next highest score a rank of 2 and so on. This eliminates the effect of outliers. It does neglect the difference in magnitude between the scores. If there are two scores that are the same, there are tied ranks. These scores are ranked by the value of the average potential rank for those scores (e.g. rank 3 and 4 will become rank 3.5).

There are several alternatives to the four most used non-parametric tests:

1. Kolmogorov-Smirnov Z
   It tests whether two groups have been drawn from the same population. It has more power than the Mann-Whitney test when the sample sizes are less than 25 per group.
2. Moses Extreme Reaction
   It tests the variability of scores across the two groups and is a non-parametric form of the Levene’s test.
3. Wald-Wolfowitz runs
   It looks at clusters of scores in order to determine whether the groups differ. If there is no difference, the ranks should be randomly interspersed.
4. Sign test
   It does the same as the Wilcoxon-signed rank test but it is only based on the direction of the difference. The magnitude of change is neglected. It lacks power unless the sample size is really small.
5. McNemar’s test
   It uses nominal, rather than ordinal data. It is useful when looking for changes in people’s scores. It compares the number of people who changed their response in one direction to those who changed in the opposite direction.
6. Marginal homogeneity
   It is an extension of McNemar’s test and is similar to the Wilcoxon test.
7. Friedman’s 2-way ANOVA by ranks (k samples)
   It is a non-parametric ANOVA to compare two groups but has low power compared to the Wilcoxon signed-rank test.
8. Median test
   It assesses whether samples are drawn from a population with the same median.
9. Jonckheere-Terpstra
   It tests for trends in the data. It tests for an ordered pattern of the medains of the group. It does the same as the Kruskal-Wallis test but incorporates the order of the groups. This test should be used when a meaningful order of medians is expected.
10. Kendall’s W
    It tests the agreement between raters and ranges between 0 and 1.
11. Cochran’s Q
    It is a Friedman test on dichotomous data.

The effect size for both the Wilcoxon rank-sum test and the Mann-Whitney test can be calculated using the following formula:

 denotes the total sample size.

WILCOXON RANK-SUM TEST
This test can be used to

Any straight line can be defined by the slope (1) and the point at which the line crosses the vertical axis of the graph (intercept) (2). The general formula for the linear model is the following:

Regression analysis refers to fitting a linear model to data and using it to predict values of an outcome variable (dependent variable) from one or more predictor variables (independent variables). The residuals are the differences between what the model predicts and the actual outcome. The residual sum of squares is used to assess the ‘goodness-of-fit’ of the model on the data. The smaller the residual sum of squares, the better the fit.

Ordinary least squares regression refers to defining the regression models for which the sum of squared errors is the minimum it can be given the data. The sum of squared differences is the total sum of squares and represents how good the mean is as a model of the observed outcome scores. The model sum of squares represents how well the model can predict the data. The larger the model sum of squares, the better the model can predict the data. The residual sum of squares uses the differences between the observed data and the model and shows how much of the data the model cannot predict.

The proportion of improvement due to the model compared to using the mean as a predictor can be calculated using the following formula:

This value represents the amount of variance in the outcome explained by the model relative to how much variation there was to explain. The F-statistic can be calculated using the following formulas:

‘k’ represents the degrees of freedom and denotes the number of predictors.

The F-statistic can also be used t test the significance of  with the null hypothesis being that  is zero. It uses the following formula:

Individual predictors can be tested using the t-statistic.

BIAS IN LINEAR MODELS
An outlier is a case that differs substantially from the main trend in the data. Standardized residuals can be used to check which residuals are unusually large and can be viewed as an outlier. Standardized residuals are residuals converted to z-scores. Standardized residuals greater than 3.29 are considered an outlier (1), if more than 1% of the sample cases have a standardized residual of greater than 2.58, the level of error in the model may be unacceptable (2) and if more than 5% of the cases have standardized residuals with an absolute value greater than 1.96, the model may be a poor representation of the data (3).

The studentized residual is the unstandardized residual divided

Moderation refers to the combined effect of two or more predictor variables on an outcome. This is also known as an interaction effect. A moderator variable is one that affects the relationship between two others. It affects the strength or direction of the relationship between the variables.

The interaction effect indicates whether moderation has occurred. The predictor and the moderator must be included for the interaction term to be valid. If, in the linear model, the interaction effect is included, then the individual predictors represent the regression of the outcome on that predictor when the other predictor is zero.

The predictors are often transformed using grand mean centring. Centring refers to transforming a variable into deviations around a fixed point. This fixed point is typically the grand mean. Centring is important when the model contains an interaction effect, as it makes the bs for lower-order effects interpretable. It makes interpreting the main effects easier (lower-order effects) if the interaction effect is not significant.

The bs of individual predictors can be interpreted as the effect of that predictor at the mean value of the sample (1) and the average effect of the predictor across the range of scores for the other predictors (2) when the variables are centred.

In order to interpret a (significant) moderation effect, a simple slopes analysis needs to be conducted. It is comparing the relationship between the predictor and outcome at low and high levels of the moderator. SPSS gives a zone of significance. Between two values of the moderator the predictor does not significantly predict the outcome and below and above the values it does.

The steps for moderation are the following if there is a significant interaction effect: centre the predictor and moderator (1), create the interaction term (2), run a forced entry regression with the centred variables and the interaction of the two centred variables (3).

The simple slopes analysis gives three models. One model for a predictor when the moderator value is low (1), one model for a predictor when the moderator value is at the mean (2) and one model for a predictor when the moderator value is high (1).

If the interaction effect is significant, then the moderation effect is also significant.

MEDIATION
Mediation refers to a situation when the relationship between the predictor variable and an outcome variable can be explained by their relationship to a third variable, the mediator. Mediation can be tested through three linear models:

1. A linear model predicting the outcome from the predictor variable (c).
2. A linear model predicting the mediator from the predictor variable (a).
3. A linear model predicting the outcome from both the predictor variable and the mediator (predictor = c’ and mediator = b).

There are four conditions for mediation: the predictor variable must significantly predict the outcome variable (in model 1)(1), the predictor variable must significantly predict the mediator

Covariates are characteristics of the participants in an experiment. These are characteristics outside of the actual treatment. If a researcher wants to compare means of multiple groups using the additional predictors, the covariates, then the ANCOVA is used. Examples of covariates could be love for puppies, softness of puppy fur.

Covariates can be included in an ANOVA for two reasons:

1. Reduce within-group error variance
   The unexplained variance is attributed to other variables, the covariates, which reduces the total error variance. This allows for a more sensitive test for the difference of group means.
2. Elimination of confounds
   By adding other variables, covariates, in the analysis, confounds are eliminated.

If there are covariates, the b-values represent the differences between the means of each group and the control adjusted for the covariate.

ASSUMPTIONS AND ISSUES WITH ANCOVA
There are two new assumptions for ANCOVA that are not present with ANOVA. These assumptions are independence of the covariate and treatment effect and homogeneity of regression slopes.

The ideal case is that the covariate is independent from the treatment effect. If the covariate is not independent from the treatment effect, then the covariate will reduce the experimental effect because it explains some of the variance that would otherwise be applicable to the experiment. The ANCOVA does not control for or balance out the differences caused by the covariate. The problem of covariates potentially explaining a bit of the data and wanting to filter these confounds is using randomizing participants to experimental groups or matching experimental groups on a covariate.

Another assumption of the ANCOVA is that the relationship between covariate and outcome variable holds true for all groups of participants and not only for a few groups of participants (e.g. for both males and females and not only males). This assumption can be checked by checking the regression line for all the covariates and all the conditions. The lines should be similar.

In order to test the assumption of homogeneity of regression slopes, the ANCOVA model should be customized on SPSS to look at the independent variable x the covariate interaction.

CALCULATING THE EFFECT SIZE
The partial eta squared is the effect size which takes the covariates into account. It uses the proportion of variance that a variable explains that is not explained by other variables in the analysis. It uses the following formula:

Repeated-measures refers to when the same entities (e.g. people) participate in all conditions of an experiment or provide data at multiple points of time.

One of the assumptions of the standard linear model is that residuals are independent, which is not true for repeated-measures designs. The residuals are affected by both between-participant factors and within-participant factors. There are two solutions to this:

1. Model within-participant variability
2. Apply additional assumptions to make a simpler, less flexible model fit

One of these assumptions is sphericity (circularity). This assumption states that the relationship between scores in pairs of treatment conditions is similar (e.g. the level of dependence between means is roughly equal). It states that variation within conditions are similar and that no two conditions are any more dependent than any other two. Local sphericity refers to when some conditions do have equal variance and some do not. Sphericity is not relevant if there are only two groups. It becomes relevant when there are at least three conditions.

The assumption of sphericity can be tested using Mauchly’s test. The degree of sphericity can be estimated using the Greenhouse-Geisser estimate or the Huyn-Feldt estimate. If the assumption of sphericity is not met, then there is a loss of power and the F-statistic doesn’t have the distribution it is supposed to have. In order to do post-hoc tests when you worry about whether the assumption of sphericity is violated, Bonferonni method can be used, if it is not violated, Tukey’s test can be used.

If the assumption of sphericity is violated, the degrees of freedom has to be adjusted. The degrees of freedom is multiplied by the estimate of sphericity to calculate the adjusted degrees of freedom.

F-STATISTIC OF REPEATED MEASURES DESIGN
In repeated measured designs, the within-groups variance consists of within-participant variance, as there is only one group of participants. This consists of the effect of the experiment and the error (variance not explained by the experiment). The between-groups variance now consists of the between-participant variance.

The formula for the within-entity (groups) variance is the following:

The n represents the number of scores within the person (e.g. number of experimental conditions). The total amount of variance that is explained by the experimental manipulation can be calculated by comparing the condition mean to the grand mean for all the conditions. It uses the following formula:

The total error variance (residual sum of squares), the amount of variance that cannot be explained by the experimental manipulation can be calculated in the following way:

In order to calculate the F-statistic, the mean squares have to be calculated and this can be done by dividing both the SSR and the SSM by the degrees of freedom:

A multivariate analysis is used when there is more than one dependent (outcome) variable. It is possible to use several F-tests when there are several dependent variables, but this inflates the type-I error rate. A MANOVA can detect whether groups differ along a combination of dimensions. MANOVA has a greater potential power to detect an effect.

A matrix is a grid of numbers arranged in columns and rows. The values within a matric are called components or elements and the rows and columns are vectors. A square matrix has an equal number of columns and rows. An identity matrix is a matrix where the diagonal numbers are ‘1’ and the non-diagonal numbers are ‘0’. The sum of squares and cross-products (SSCP) matrices are a way of operationalize multivariate versions of the sums of squares. The matrix that represents the systematic variance (model sum of squares) is denoted by the letter ‘H’ and is called the hypothesis sum of squares and cross-products matrix (hypothesis SCCP). The matrix that represents the unsystematic variance (residual sum of squares) is denoted by the letter ‘E’ and is called the error sums of squares and cross-products matrix (error SSCP). The matrix that represents the total sums of squares for each outcome (total SSCP) is denoted by the letter ‘T’.

The cross-product is the total combined error between two variables.

THEORY BEHIND MANOVA
The total sum of squares is calculated by calculating the difference between each of the scores and the mean of those scores, then squaring those differences and adding them together.

The degrees of freedom is N-1. The model sum of squares is calculated by taking the difference between each group mean and the grand mean, squaring it, multiplying by the number of scores in the group and then adding it all together.

The degrees of freedom is the sample size of each group minus one multiplied by the number of groups. The SST and the SSM then have to be divided by their own degrees of freedom, before being divided by each other to get to the F-statistic.

The cross-product is the difference between the scores and the mean for one variable multiplied by the difference between the scores and the mean for another variable. It is similar to covariance. It uses the following formula:

For each outcome (dependent) variable, the score is taken and subtracted from the grand mean for that variable. This gives x values per participant, with x being the number of outcome variables.

The model cross-product, how the relationship between the outcome variables is influenced by the experimental manipulation, uses the following formula:

The residual cross-product, how the relationship between the outcome variables is influenced by individual differences and unmeasured variables, can be calculated

It is possible to predict categorical outcome variables, meaning, in which category an entity falls. When looking at categorical variables, frequencies are used. The chi-squared test can be used to see whether there is a relationship between two categorical variables. It is comparing the observed frequencies with the expected frequencies. The chi-squared test standardizes the deviation for each observation and these are added together.

The chi-squared test uses the following formula:

The expected score has the following formula:

The degrees of freedom of the chi-squared distribution are (r-1)(c-1). In order to use the chi-squared distribution with the chi-squared statistic, there is a need for the expected value in each cell to be greater than 5. If this is not the case, then Fisher’s exact test can be used.

The likelihood ratio statistic is an alternative to the chi-square statistic. It is comparing the probability of obtaining the observed data with the probability of obtaining the same data under the null hypothesis. The likelihood ratio statistic uses the following formula:

It uses the chi-squared distribution and is the preferred test if the sample size is small. The chi-square statistic tends to make a type-I error if the table is 2 x 2. This can be corrected for by using Yates’ correction and uses the following formula:

In short, the chi-square test tests whether there is a significant association between two categorical variables.

ASSUMPTIONS WHEN ANALYSING CATEGORICAL DATA
One assumption the chi-square test uses is the assumption of independence of cases. Each person, item or entity must contribute to only one cell of the contingency table. Another assumption is that in 2x2 tables, no expected value should be below 5. In larger tables, not more than 20% of the expected values should be below 5 and all expected values should be greater than 1. Not meeting this assumption leads to a reduction in test power.

The residual is the error between what the expected frequency and the observed frequency. The standardized residual can be calculated in the following way:

Individual standardized residuals have a direct relationship with the test statistic, as the chi-square statistic is composed of the sum of the standardized residuals. The standardized residuals behave like z-scores.

EFFECT SIZE
Cramer’s V can give an effect size. In 2x2 tables, the odds-ratio is often used as the effect size. The odds-ratio uses the following formula:

The actual odds ratio is the odds of event A divided by the odds of event B.