Factor analysis and principal component analysis (PCA) are techniques for identifying clusters of variables. These techniques have three uses: understanding the structure of a set of variables (1), construct a questionnaire to measure an underlying variable (2) and reduce a dataset to a more manageable size while retaining as much of the original information as possible (3).

Factor analysis attempts to achieve parsimony by explaining the maximum amount of common variance in a correlation matrix using the smallest number of explanatory constructs (latent variables). PCA attempts to explain the maximum amount of total variance in a correlation matrix by transforming the original variables into linear components.

A factor loading refers to the coordinate of a variable along a classification axis (e.g. Pearson correlation between factor and variable). It tells us something about the relative contribution that a variable makes to a factor.

In factor analysis, scores on the measured variables are predicted from the means of those variables plus a person’s scores on the common factors (e.g. factors that explain the correlations between variables) multiplied by their factor loadings, plus scores on any unique factors within the data (e.g. factors that cannot explain the correlations between variables).

In PCA, the components are predicted from the measured variables.

One major assumption of factor analysis is that the algebraic factors represent real-world dimensions. A regression technique can be used to predict a person’s score on a factor. Using this technique, the resulting actor scores have a mean of 0 and a variance equal to the squared multiple correlation between the estimated factor scores and the true factor values. A downside is that the scores can correlate with other factor scores from a different orthogonal factor. The Bartlett method and the Anderson-Rubin method can be used to overcome this problem. The Bartlett method produces factor scores that are uncorrelated and standardized.

DISCOVERING FACTORS
The method used for discovering factors depends on whether the results should be generalized from the sample to the population (1) and whether you are exploring your data or testing a specific hypothesis (2).

Random variance refers to variance that is specific to one measure but not reliably so. Communality refers to the proportion of common variance present in a variable. Extraction refers to the process of deciding how many factors to keep.

Eigenvalues associated with a variate indicate the substantive importance of that factor. Therefore, factors with large eigenvalues are retained. Eigenvalues represent the amount of variation explained by a factor.

A scree plot is a plot where each eigenvalue is plotted against the factor with which it is associated. The point of inflexion is where the slope of the line changes dramatically. This point can be used as a cut-off point to retain factors. It is also possible to use eigenvalues as a criterion. Kaiser’s criterion is to retain factors with eigenvalues greater than 1. Joliffe’s criterion is to retain factors with eigenvalues greater than 0.7.

It is possible to calculate the degree to which variables load onto extracted factors, although interpretation is difficult. In order to make interpretation easier, rotation can be used where rotation rotates aces such that variables are loaded maximally to only one factor. Orthogonal rotation refers to rotation while keeping the factors uncorrelated. Oblique rotation allows factors to correlate.  

The choice of orthogonal or oblique rotation depends on whether there is a theoretical reason to suppose that the factors should correlate or should be uncorrelated (1) and how the variables cluster on the factors before rotation (2).

The reliability of factor analysis depends on sample size. If correlations are too high or too low, the variables should be removed from the analysis. The determinant tells us whether the correlation matrix is singular or if all variables are completely unrelated. The determinant should be larger than 0.00001.

A non-positive definite matrix is not possible. The most likely reason for this is having too many variables and too few cases of data. Variables in diagonal line on the anti-image matrix with a score of less than 0.5 should be removed. These scores denote the Kaiser-Meyer-Olkin measure of sampling adequacy. The off-diagonal scores should be small.

The first part of factor extraction is to determine the linear components within the variables – the eigenvectors. With oblique rotation, the factor matrix is split into the pattern matrix and the structure matrix. The pattern matrix contains the factor loadings. The structure matrix adjusts for the relationship between factors. It is a product of the pattern matrix.

Reliability means that a measure should consistently reflect the construct that it is measuring. Split-half reliability splits the scale set into two randomly selected set of items. A score for each participant is calculated on each half of the scale. If a scale is reliable, a person’s score on one half of the scale should be the same as their score on the other half.

For cognitive tests, Cronbach’s alpha of 0.8 suffices. For ability tests, Cronbach’s alpha of 0.7 suffices. The alpha depends on the number of items on the scale. It also should not be used as a measure of unidimensionality. Lastly, reverse-phrased items should be scored the other way around.