In chapters 4 and 8 we discussed the internal structure (i.e., dimensionality) of a psychological test. As we briefly introduced there, the internal structure of a test has to do with the number and nature of the psychological constructs that we measure with the items. One way to identify those constructs is through factor analysis. In this chapter, we will discuss factor analysis, and in particular confirmatory factor analysis (CFA).

What are EFA and CFA used for?

There are two types of factor analysis: exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). These two types of factor analysis are most suitable for different phases of test development and evaluation. EFA is most suitable for the first phases of test use (clarifying the construct and the test). CFA is most suitable in later phases of test use, after the initial evaluations of item properties and dimensionality and after major revisions of the test content (ie when the test content is virtually fixed). Confirmative factor analysis (CFA) is used to investigate the dimensionality of a test when there are already hypotheses about the number of underlying factors (dimensions), the connections between items and factors, and the coherence of the factors.

What is the purpose of CFA?

With CFA we evaluate hypotheses about the internal structure or dimensionality of a measurement model. CFA shows the extent to which the assumed measurement models correspond to the actual data of the respondents. Thereafter, if required, the assumed model can be adjusted to better match the actual data.

How do you perform a CFA?

After a specific measurement model has been evaluated, the model is usually adjusted and then the adjusted model is evaluated again using CFA. A model is often adapted and evaluated several times.

Before you perform a CFA, there are three important things you must do. First, you must make clear which psychological construct you are going to measure and develop a number of test items. Secondly, you must find enough people to take the test. Finally, all items must have the same direction, so you must score negatively coded items in reverse. Performing a CFA consists of four steps:

1. specification of the measurement model;
2. calculations;
3. interpret and report the results;
4. model changes and new analysis (if necessary). 

These four steps are discussed below.

Step 1: specification of the measurement model

Enter the data in a statistical software program. You make a figure of the measurement model and the program then converts it into formulas. First the number of dimensions (also called factors or latent variables) must be determined. It must then be determined which items are linked to which factors. At least one item is associated with each factor. And each item is usually connected to only one latent variable. If a model is multidimensional, then it must also be determined which factors may be associated with other factors. We only need to determine whether or not there are connections, the software will then estimate the precise values ​​of these connections.

Step 2: calculations

After we have entered all the details of the measurement model, we have the program run a CFA. Although these calculations are performed 'behind the scenes', it is still useful to know the statistical process. The basic calculations have four phases:

1. The data is used to calculate the actual item variances and covarities between items.
2. The actual variances and covarities of the items are used to estimate the parameters. There are several important parameters. One is the factor loading(s) of each item. This is the extent to which an item is associated with a factor. A second parameter is the connections between different factors. CFA also calculates the significance of each parameter.
3. The estimated parameter values ​​are used to calculate implied item variances and covarities. So the program calculates item variances and covarities as they are implied by the estimated parameters. If the assumed model is correct then the implied variances and covarities correspond to the actual variances and covarities from the first step.
4. The software program provides information regarding the general suitability or "fit" of the assumed model. It compares implied variances / covarities with actual variances / covarities and it calculates a 'model fit' and 'adjustment indexes' ( modification indices). These adjustment indexes provide specific ways in which the measurement model could be improved.

Step 3: interpret and report the results

After entering the data and calculating parameters and the 'fit' of the model, the results are interpreted.

First we look at the fit of the model. A 'good fit' ( ' good fit') indicates that the supposed model matches the actual responses to the test, this supports the validity of the model. A 'bad fit' ( ' poor fit') indicates that the assumed number of dimensions does not correspond to the actual responses to the test. The chi-square is a measure that is used to indicate the degree of ' poorness of fit' of the model. Large, significant chi-squared values ​​indicate a poor fit, and small, non-significant chi-squared values ​​indicate a good fit of the model. Sample size influences the chi-square. A large sample provides large chi-squared values, which in turn provide statistical significance. In addition to the chi-square, a CFA provides a number of other fit indexes. These indexes do not produce statistical significance and all of these indexes have different scales and norms.

If the fit indexes indicate that the model is not suitable, then the adjustment indexes are viewed and it is looked into how the model could be improved. If the fit indexes indicate that the model is suitable, then the parameter estimates are viewed.

If the hypothesis is that an item is associated with a certain factor, then we expect to find a large, positive, and statistically significant factor load. If we find that, then the item is a good reflection of the underlying psychological dimension. And we keep this item in the test. If the factor load is small and / or not significant, the item is not related to the psychological dimension and the item is removed from the test. Then the model is adjusted and all calculations are done again.

Step 4: model change and new analysis (if required) 

If the model is not suitable, then we switch to viewing the adjustment indexes and adjusting the assumed measurement model. An adjustment index indicates the potential influence of adjusting a specific parameter. After modifying the model, it is analyzed again, so all calculations are done again.

How can CFA be used to evaluate reliability?

CFA is also sometimes used as a method to estimate reliability. First we use CFA to evaluate the basic measurement model of the test. Then, if necessary, we adjust the measurement model and re-analyze it. Finally, we use the non-standardized parameter estimates to estimate the reliability of the test:

Reliability = true variance / (true variance + error variance)

So, estimated reliability = (∑גi)² / (((∑גi)² + ∑өii + 2∑өij) 

גi = factor loading of an item.

Өii = error variance of an item.

Өij = covariance between the errors of two items.

(∑גi)² = is the variance of the true scores. 

Σөii + 2Σөij = the random error variance.

How can CFA be used to evaluate validity?

CFA can also evaluate validity in various ways. Firstly, CFA provides insight into the 'internal structure' aspect of validity. Second, if responses to a test are measured together with measurements of related constructs or criteria, then we can evaluate the relationship between the test and those variables. This provides important information about the psychological significance of the test scores. There are two ways we can use CFA to view these validity components. We can use CFA to evaluate convergent and discriminant validity by applying CFA to multitrait-multimethod matrices. In addition, we can evaluate convergent validity by examining a test and one or more criterion variables using CFA.

How can CFA be used to assess 'measurement invariance'?

CFA has also recently been used to evaluate group differences in the psychometric properties of tests. CFA is particularly useful in the conceptualization and detection of construct bias ("measurement invariance"). Construct bias implies that the test has a different internal structure, and therefore meaning, for different groups (see Chapter 11 for a more detailed description). Vice versa, if the internal structure of a test does not differ between groups, then this evidence is against construct bias, and we speak of an internal structure in variance.

Measurement invariance can be mapped with CFA by comparing groups in terms of specific parameters (such as the lambda , the theta , etc.) of measurement models. If groups have different values ​​for a parameter, then this is proof of a lack of invariance for the parameter (and therefore proof of a certain degree of construct bias, because the parameters differ between groups). The extent to which there are differences can be summarized in four different levels of measurement invariance:

1. Configural;
2. weak/metric; 
3. strong/scalar; 
4. strict. 

In short, the greater the difference, the less robust you st test for measurement invariance (the first level is the weakest, least robust level).  

In chapters 4 and 8 we discussed the internal structure (i.e., dimensionality) of a psychological test. As we briefly introduced there, the internal structure of a test has to do with the number and nature of the psychological constructs that we measure with the items. One way to identify those constructs is through factor analysis. In this chapter, we will discuss factor analysis, and in particular confirmatory factor analysis (CFA).