Summaries per article with Psychometrics at Leiden University 21/22

What is principal component analysis (PCA) and what is its purpose?

Principal component analysis (PCA) is a multivariate data analysis technique used to analyze a large number of variables simultaneously. The analysis is concerned with the structure that the relationships between the variables take. The main goal is data reduction. PCA is often confused with factor analysis (FA), this distinction will be discussed later. The Purpose of PCA The aim is to get an idea of the most important associations between the items of, for example, a questionnaire. In this way, subgroups are obtained that strongly correlate with each other. These subgroups are known as components or factors. PCA is also important for the making of questionnaires. If subgroups correspond with each other we can combine them to create questionnaires or scales.

What does PCA do?

In particular, PCA is a tool for data reduction in which researchers try to lose as little information about the variables as possible. The information used in PCA is the association between the observed variables, expressed in correlations or variances and covariances. After these correlations are created the most important associations are combined into components. We can show this in different ways. The first way is geometrical. Here, two correlated variables are shown through two components that are not correlated. Thus, each component visualizes a variable and is built up from two observations from the same person on this variable. In practice, PCA is much more interesting with more variables, but we cannot show this geometrically anymore. This is why we can also show PCA algebraically. In this case, we use squares to represent the observed variables (Xi) and circles to show the components (Fj). We also try to represent the weight of the different variables and components (aij). The weight is also known as component charges. For example, if we want to show two components and five observed variables, the comparison of the two components (F1 and F2) looks like this:

F1 = a11X1 + A21X2 + a31X3 +a41X4 + a51X5

F2 = a12X1 + a22X2 + a32X3 + a42X4 + a52X5

These equations show that the components Fj are calculated as linear combinations of the observed variables Xi. The weights aij are also shown, in which the i refers to the observed variable and the j to the component. F1 is as equal as possible to all X variables. Components can also be considered variables themselves, and can be used as variables in other data analysis techniques. Component scores can thus be used to examine individual differences.

How can we explain variance with PCA?

In PCA, the correlation between the variables and components is maximized. The variables and components also have common variance and because we maximize the correlation, this variance is also maximized. Because of this we can predict how much of the variance of a certain variable can be explained by a component. All components together can explain a certain amount of the variance. In PCA the observed variables are usually standardized.

The component charges aij are equal to the Pearson correlations between Xi and Fj. When we square the charge we get the proportion of variance of Xi that is explained by Fj. The amount of variance of a certain group of observed variables a component can explain is known as eigenvalue. The eigenvalue is equal to the sum of the squared factor charges of all X variables that are part of that component. This is mathematically shown as λj Σpi=1 aij2. The eigenvalue of the first component is always the largest, then follows the second component, etc. Dividing the eigenvalue by the number of variables gives you the proportion of explained variance.

The amount of variance of variable Xi that can be explained by the component is known as the communality of that variable. This is mathematically shown as h². When we sum the squared component charges of variable Xi of the components gives the communality of that variable, mathematically shown as: hi2 Σkj=1 aij2. Communality is also used as a measure of fit: how well a variable fits into a factor sollution.

If there are as many components as there are variables, 100% of the variance is explained by the components. This is known as full dimensionality. Full dimensionality means however that we cannot speak of data reduction. We also have a rotation problem: the specific system of coordinates does not matter for the solution, meaning there are an infinite number of ways the describe the same solution.

What do we do before we do PCA?

PCA does not always provide a useful solution. The main threats are that PCA solutions may be unstable (varying widely from sample to sample) or merely display randomness. Therefore, preliminary checks must be made.

First, we need to protect against randomness or arbitrariness. We can do this with Bartletts test. This test checks whether the correlations between the variables in the analysis are zero. We want this test to be statistically significant. However, with real psychological data this test always is. This is why the best indication of usability is the Kaiser-Meyer-Olkin (KMO) measurement. If the data has a clear factor structure, the partial correlations between pairs of variables are very close to zero. It is calculated as follows: sum of squared correlations / (sum of squared correlations + sum of squared partial correlations). KMO usually has a value between 0.5 (worst case) and 1.00 (best case, when all partial correlations are zero). Values ​​higher than 0.7 are good enough.

Second, we need to get stable results that do not vary from sample to sample. We can do this by getting a big enough sample. Rules of thumb:

• Sample size: a sample equal to or bigger than 300 is almost always big enough.
• Factor loadings: if a factor has four or more loadings with an absolute value greater than 0.6, the sample size doesn't matter. Factors with at least ten loadings greater than 0.4 are stable when N is greater than 150.
• Communalities: if (almost) all communalities are greater than 0.6, a sample size of 100 is good, and above 0.5 a sample size between 100 and 200 is good enough.

How many components does PCA use and how can we decide this?

Finding the factors or components is called factor extraction. With more factors, more variance is explained, but the solution is less useful. There is no universal criterion to solve this problem, but four criteria can be used:

• The eigenvalue-greater-than-one criterion: the eigenvalue must be at least one, because each variable has a variance of one.
• Interpretability: components must be meaningful. We determine this on the basis of general knowledge, theory or knowledge of previous research.
• The elbow criterion: according to this criterion, you must stop extracting factors at the point in the eigenvalue graph where the curve resembles an elbow. Sometimes no clear 'elbow' can be found, or several can be seen. The advice is to try j-1 and j+1 components, and choose the one with the clearest interpretation.
• The 'crushed stone' criterion: sometimes there are points in the eigenvalue graph through which a clear straight line can be drawn. The criterion says that the solution must contain only those components whose eigenvalues ​​are above this straight line. The problem is that there is not always a clear straight line.

It is advised to always make a graph with SPSS of the eigenvalues. If an 'elbow' is visible, this is always the best method (plus or minus 1).

How does rotation and interpretation work?

A PCA solution can be described in an infinite number of ways, which are mathematically similar, but can lead to very different interpretations. This is called the rotation problem. The rotation problem arises from changing the perspective from which we look at the solution. For example, if we rotate F1 and F2 30 degrees to the right, new X and Y axes (F1' and F2') are created. With this, new coordinates for each vector (or variable) can be calculated. This leads to different interpretations.

The aim of doing such a rotation is to find an understandable and therefore simple solution. A simple solution has as little components as possible and each component must be related to a small number of observed variables. Such a simple solution is almost impossible because the components are chosen in such a way that the first component explains as much variance as possible, then the second component also has to explain as much variance as possible and so on. This is good for data reduction, but not a simple solution. PCA is biased towards finding a common first component. This common first component highlights what all observed variables have in common. All subsequent components are contrast components, because they are orthogonal to the first component. Especially in situations with many items and components, this solution is complex. Therefore, a rotation that finds a simple structure in the data is desirable. This can be done in various ways.

The first rotation method is orthogonal rotation or VARIMAX rotation. The VARIMAX rotation attempts to maximize the variance of the loadings for each component. If there is a simple structure in the data (homogeneous subsets of items that are not strongly correlated with items of other subsets), VARIMAX finds this with more ease than the other rotations. After VARIMAX, the communalities remain the same. The total explained variance also remains the same. The new coordinate system remains orthogonal (at an angle of 90 degrees). The second rotation method is known as non-orthogonal rotation, oblique rotation or OBLIMIN rotation. This is a rotation method where the angles of the axes change. This provides a more realistic interpretation, but it does get more complicated. Therefore, and for historical reasons, VARIMAX is the most widely used. Always do a VARIMAX rotation, but if this solution is incomprehensible, the unrotated solution should be looked at.

What are the most important SPSS settings needed?

• Method of analysis: in the extraction screen within 'factor' we can choose to do a PCA.
• The amount of components: can also be chosen in the extraction screen.
• The unrotated solution and eigenvalue plots: can be found by checking 'unrotated factor solution' and 'Scree plot'.
• Rotation method: in the rotation screen we can select 'VARIMAX'
• The sorting of variables: we can choose 'sorting by size' in the options screen to sort variables from high to low charges on each component.

Where can we find eigenvalues, communalities and explained variance in SPSS?

Under the heading 'extraction sum of squared loadings' in SPSS are the eigenvalues and the explained variance for the unrotated components. If there is a strong data reduction, the explained variance is usually not too high. The explained variance and the eigenvalues for the rotated components can be found under the heading 'rotation sum of squared loadings'. The rotation does not change the total explained variance, but the variance is more evenly distributed among the components. If a variable has a low communality, it is a unique variable. This could mean that the variable is measuring something completely different than we want, but it could also mean that the item is measuring something important, which is not measured by other items.

What are the assumptions of PCA?

PCA has three assumptions:

• The relationships between the observed variables are linear. We can check this by using scatterplots.
• The observed variables follow a multivariate bell-curve. This assumption is not the most important one to stick to.
• The correlations between the observed variables must be reliable. To get to this the sample must be representative and big enough.

What is the difference between PCA and factor analysis (FA)?

PCA is an empirical summary of data. This is because we find components by adding observed variables with a certain weight. The components represent the observed variables, and as such PCA is a summary. It wants to figure out what part of the variance of the variables is important and which part is not.

FA on the other hand, wants to discover what is common in the variance for variables in comparison to what is unique for individual variables. In FA a model is made for the observed associations within a set of variables. This model tries to reproduce the associations as closely as possible. The factors in this model, which are also known as latent variables, are hypothetical constructs that we can only measure indirectly, which is not the case for PCA. In FA, a unique factor is also created for each variable. This unique factor explains all the variance of that variable which could not be explained by the general factors. The unique factors thus show error. FA is sometimes called a structural or causal model.

The factor analysis model that FA uses specifies each observed variable as the weighted sum of the common factors and a unique factor specific to that variable. In FA, the factors are the cause of the observed correlations, unlike in PCA. The second difference is the presence of unique factors (Uj). Unique factors are not correlated with each other or with general factors. FA and PCA often come to the same conclusion because they use the same information. Results may differ only for a small number of variables. FA is mainly a model for the observed correlations, while PCA is a technique that tries to explain as much variance as possible.

In summary, we can say that PCA wants to explain the variances, while FA wants to explain the covariance of the observed variables.

Psychometric research is conducted in the hope of making meaningful statements about ourselves and others. There are two kinds of judgments we can make. First, there are dimensional judgments, which look at our position compared to others on a particular dimension. In addition, we can also use classification, meaning we determine to which category we belong.

How can we go from a dimension to a classification?

There are several ways we can categorize people. The general procedure for moving from a dimensional judgment to a classification starts with a research group. This group is a sample for which we know the scores on each dimension for each individual. Within such a sample, an attempt is made to determine to which group an individual belongs based on a dimensional judgment. Although we actually already know the classification, this ensures that we have a prediction rule that we can use for new individuals and that we have information about how well the prediction rule works. Then, if we find that the prediction rule works well enough, we can use it to classify new individuals from which we do not know to which group they belong.

Although the process sounds simple in this way, in practice it does not work very easily. For example, we often doubt the reliability and validity of a sample or this particular procedure, but we still base our prediction rule on this. Next, there are different criteria for determining how accurate a prediction is, which can contradict each other. Lastly, applying a prediction rule to a new group can lead to unexpected results.

How does the basic process of classification go?

We can make the easiest dimensional classification when we look at two groups and base our classification on just one dimension. With this data we can perform a t-test to see if there is a significant difference between the means of the groups. However, by showing that a certain interval variable (such as depression) is related to a nominal variable with two categories (depression status) we have not solved the classification problem. We do not want to predict from nominal to interval variables, but from interval to nominal variables. If only one interval predictor is used, a cut-off point can be established. A cut-off point is used to determine above which point people receive a positive diagnosis, and below which point they will receive a negative diagnosis.

In reality, this prediction rule will not work perfectly because there is overlap between the distributions of the two groups on the interval variable. This means that we can always make two kinds of mistakes. The first one is a false positive, where for example someone does not have depression but is classified as having it. The second one is false negatives, where for example depressed people are classified as not depressed.

How do we determine the cut-off score?

Which cut-off score we use depends on the extent to which we consider both types of errors bad. If we find both types of error equally bad and the groups have the same symmetric distribution with the same standard deviation, the point will be exactly between the two group means. If we find false positives worse and want to eliminate them, we run the risk of finding more false negatives and vice versa. This shows that we always have to make certain choices if we want to determine the cut-off score. The situation becomes even more complex if we want to compare more than two groups on different dimensions. In that case, a discriminant analysis is often performed.

What is discriminant analysis (DA)?

The aim of discriminant analysis is to predict as accurately as possible to which group a certain person belongs by using a certain number of interval variables (>2). We can look at the differences in two ways: from the group perspective and the individual perspective. In the first case, we try to describe the nature of the differences between groups, which is called descriptive discriminant analysis. In addition, we can also take the individual as a starting point and use the scores on the interval variables to predict to which group the person belongs, which is known as predictive discriminant analysis. Here, the emphasis is on the latter variant.

The first question to ask is whether our prediction is meaningful. DA leads to an optimal (best possible) prediction of the nominal variable based on the interval variables. To see if the prediction is meaningful, you can check whether the best possible prediction is better than you would expect at the probability level, using Wilk's Lambda. If this test is not significant, we cannot say anything useful about which group someone would belong to based on the interval variables, and the prediction is not meaningful. It is important to note that a significant result does not guarantee an accurate prediction. You sometimes find several Wilk's Lambdas in SPSS output: in that case you have to use the top one, as this is more accurate.

In the context of a predictive DA, you don't need to know how groups differ (if individual classification is your goal). However, as psychologists, we often also want to know how and why these predictions work. A rough, but fairly effective method is to compare the means on the interval variables. A major shortcoming of this approach is that it does not take into account intercorrelations between predictors, which can lead to misleading conclusions. To solve this problem, you can use descriptive discriminant analysis.

Which problems do we need to solve in order to predict to which group an individual belongs?

Calculating the most likely group membership for each possible individual is a problem that does not have a particular optimal solution that is best in all situations. A possible strategy for this is to look at both individuals and group means on p variables in a p-dimensional space. In this space we can calculate the differences between each individual point and all group means (using the Pythagorean theorem). Then each individual is counted as part of the group to which this person has the shortest distance. You can also apply this method if there are more than two variables, although you can no longer represent it in spatial terms if you use more than three variables.

In order to be able to use an effective method to predict which group an individual belongs to, a number of problems must be solved:

• If there are differences in standard deviations (SD) between variables, variables with a high SD will disproportionately affect the calculated distances. The solution to this problem is standardization (through Z-scores).
• When variables are correlated with each other, the variance that the variables share has a disproportionate impact on the distances, even if all variables are standardized. The solution to this problem is to work within a standardized component space or in the space of discriminant function variates.
• If there are differences between groups in variability around the mean, homogeneous groups will require a shorter distance from the group than heterogeneous groups. This can be solved by weighting the distances of certain group points based on the group their SDs.
• The boundaries between the groups do not necessarily have to be linear. With linear DA, this cannot be discovered and cannot be used as an optimal classification for individuals.

How do we determine how accurate a prediction is?

To determine how accurate a DA's prediction is, a classification table is used. This is a table in which the predicted and observed values ​​are plotted against each other, the cells of which contain the frequencies of all possible combinations. A general measure of the quality of the prediction is the percent accuracy in classification (PAC). This is the number of correct predictions / total number of predictions. In many cases, a general measure such as the PAC is not precise enough, because all errors are looked at together. With regard to more specific measures of the quality of the prediction, a distinction can be made between the quality of the instrument (sensitivity and specificity) and the quality of the individual diagnosis (positive and negative predictive value).

How do we determine the quality of an instrument?

In order to determine the quality of an instrument, it is important to determine how likely it is that an individual from a certain group will also be identified as a member of that group, which is called sensitivity. This is the number of good predictions sick in group A / total number of predictions sick in group A. A high degree of sensitivity increases false positives and decreases specificity and true negatives. Another measure of the quality of an instrument is specificity: the number of accurate predictions not ill in group A / total number of predictions not ill in group A.

Both sensitivity and specificity are conditional probabilities. This refers to the probability of event A if we know that another event (B) has occurred.

How do we determine the quality of individual diagnosis?

If you want to make a diagnosis for a certain individual, sensitivity and specificity are not useful to assess the quality. This is because you do not want to go from real situation (Y) to the conditional probability of a specific diagnosis (X), but from a certain diagnosis (X) to the conditional probability of a real situation. Instead, you can use the positive predictive value. This is the percentage of individuals with a positive diagnosis who also belong to the target group. You can also use the negative predictive value, meaning the percentage of individuals with a negative diagnosis who also belong to the target group. These concepts can all be used with multiple groups, which also offers the opportunity to answer multiple questions.

What is Bayes' rule?

If we subject a test battery to a predictive discriminant analysis, it is nice to work with groups of approximately equal size, because our predictions will have maximum precision and statistical power. In reality, groups are usually not equal in the population.

When you move from the original study group to the population, the sensitivity and specificity do not change, but the positive and negative predictive values do. If the distribution is skewed (e.g. a disease is very rare) there will be more false positives than true positives. To take this into account, you can use Bayes' theorem.

In discriminant analysis it is important to take the relative seriousness of the errors that can be made into account, as well as the relative frequencies of the groups to be predicted in the population, known as the base rate.