How does multiple regression with both quantitative and categorical predictors work? – Chapter 13

13.1 What do models with both quantitative and categorical predictors look like?

Multiple regression is also feasible for a combination of quantitative and categorical predictors. In a lot of research it makes sense to control for a quantitative variable. A quantitative control variable is called a covariate and it is studied using analysis of covariance (ANCOVA).

A graph helps to research the effect of quantitative predictor x on the response y, while controlling for the categorical predictor z. For two categories, z can be the dummy variable, else more dummy variables are required (like z₁ and z₂). The values of z can be 1 ('agree') or 0 ('don't agree'). If there is no interaction, the lines that fit the data best are parallel and the slopes are the same. It's even possible that the regression lines are exactly the same. But if they aren't parallel, there is interaction.

The predictor can be quantitative and the control variable can be categorial, but this can also be the other way around. Software compares the means. A regression model with three categories is:: E(y) = α + βx + β₁z₁ + β₂z₂, in which β is the effect of x on y for all groups z. For every additional quantitative variable a βx is added. For every additional categorical variable a dummy variable is added (or several, depending on the number of categories). Cross-product terms are added in case of interaction.

13.2 Which inferential methods are available for regression with quantitative and categorical predictors?

The first step to making predictions is testing whether a model needs to include interaction. A F-test compares a model with cross-product terms to a model without. For this the F-test uses the partial sum of squares; the variability in y that is explained by a certain variable when the other aspects are already accounted for. The null hypothesis says that the slopes of the cross-product terms are 0, the alternative hypothesis says that there is interaction. In a graph, interaction looks like this:

Another F-test checks whether a complete or a reduced model is better. To compare a complete model (E(y) = α + βx + β₁z₁ + β₂z₂) with a reduced model (E(y) = α + βx), the null hypothesis is that the slopes β₁ and β₂ both are 0. The complete model consists of three parallel lines, the reduced model only has one line. When P is small, then there is much evidence against the null hypothesis and then the complete model fits the data significantly better. The multiple coefficient of determination R² indicates how well the possible regression lines predict y and helps compare the complete with the reduced model. In a graph:

13.3 In what kind of case studies is multiple regression analysis required?

Case studies often start with the desire to research the effect of an explanatory variable on a response variable. Throughout the research, predictors are added, sometimes confounding predictors, sometimes mediating predictors.

13.4 How do you use adjusted means?

An adjusted mean or least squares mean is the mean of y for a group while controlling for the other variables in the model. The other variables are kept at a mean, so the value of the adjusted mean can be researched. When an outlier has too big of an influence on the mean, this outlier can be left out and the adjusted mean can be calculated.

The adjusted mean is indicated with an accent. The adjusted sample mean of group i is:

The coefficients equal the differences between the adjusted means. Due to the adjusted mean, the regression line of the sample mean shifts upward or downward. The Bonferroni method allows multiple comparisons of adjusted means using confidence intervals with a shared error rate.

Adjusted means are less appropriate if the means for x are very different. Using adjusted means only should be done if it makes sense that certain groups would be distributed in a certain way and if the linear shape is unchanged.

13.5 What does a linear mixed model look like?

Factors with a limited number of outcomes (like vegetarians, vegans and meat eaters) are fixed effects. Random effects on the other hand are factors of which the outcomes happen randomly (like the characteristics of research subjects). Linear mixed models have explanatory variable with both fixed effects and random effects.

A regular regression model can express the equation per subject, for instance with the value x_i1 of variable x for subject i: y_i = α + β₁x_i1 + β₂x_i2 + … + β_px_ip + ϵ_i. The error term ϵ is the variability of the responses of subjects for certain values of the explanatory variables. The sample value of this is the residual for subject i. Because the error term is expected to be 0, it is removed from the equation of E(y_i).

A linear mixed model can handle multiple correlated observations per subject: y_ij = α + β₁x_ij1 + β₂x_ij2 + … + β_px_ijk + s_i + ϵ _ij. In this y_ij is observation j (at a certain time) of subject i. For variable x₁ the observation j of subject i is written as x_ij1 and a random effect of subject i is s_i. A subject with a high positive s_i has relatively high responses for each j. The fixed effects are the parameters (β₁ etc).

The structure gives information about the character of the correlation in the model. When the correlations between all possible pairs of observations of the explanatory variables are equal, there is compound symmetry. When in longitudinal research the observations are more correlated around the start, it's an autoregressive structure. When assumptions about the pattern of correlation are best avoided, it's called unstructured. An intraclass correlation means that subjects within a group are alike. The random effects aren't just subjects, they can also be clusters of similar subjects.

The advantages of linear mixed models compared to repeated measures ANOVA is that they make less assumptions and that the consequences of missing data are less severe. When data is missing randomly, bias doesn't need to happen. Linear mixed models can be extended and twisted in all sorts of ways, even for special kinds of correlation.