What is the generalizability theory? - Chapter 13

The Generalizability Theory (G theory) helps us to distinguish the effects of multiple facets and then to use different measurement strategies. It is an ideal framework for complex measurement strategies in which several facets influence the measurement quality. This is a fundamental difference compared to the classical test theory (CTT), where different facets are not assumed.

What is a facet and what role do facets play in the complexity of a measurement strategy?

According to the G theory, measurement errors can be differentiated in different facets.

G theory can be used to investigate what effect the various aspects of a measurement strategy have on the total quality of the measurement. In this way the various items can also be examined. For example, when investigating which items are related to the onset of aggression each combination of items can be investigated separately. Each part of the measurement strategy is called a facet and different measurement strategies are partly defined by the number of facets. The more facets a measurement strategy has, the more complex the strategy is. An example of three facets is: items, observers and situations.

Which two key components play a role in G theory?

The concept of generalizability, as the name suggests, is very important within the G theory. The measurement quality is usually evaluated in terms of the ability to draw conclusions from a limited number of observations to an unlimited number of observations. When a psychological or behavioral variable is observed, only a limited number of observations can be made. The aim of the G theory is to obtain scores that are representative of the scores that would have been obtained if all possible items that could measure the construct were used.

The concept of consistency is also very important within the G theory. It is important to see whether the degree of variability of an individual's test scores is consistent with the variability of universal scores. In the G theory, estimates of generalizability are based on variance components which represent the extent to which differences exist within the 'universe' for each element of the design. A variance component is the variance of universal scores within the population of individuals. The magnitude of the variance component of a facet indicates the extent to which the facet influences observed scores.

Which two phases are distinguished in the G theory?

The G theory can be used for multiple types of analysis, but a basic psychometric analysis consists of a two-phase process: the G study and the D study. The variance components are estimated in the first phase. In such a study, factors are identified that influence the observed variance (and therefore the generalizability). This phase is called a G study, because it is used to identify to what extent the different facets could influence generalizability. In the second phase, the results of phase one are used to estimate the generalizability of the different combinations of facets. This phase is known as a D study, because the phase is used to make decisions about future measurement strategies.

The various steps associated with the two studies are discussed in more detail below.

1. G study

In this phase, variance analysis (ANOVA) is used to generate estimates of variance components for each factor. The purpose of ANOVA is to investigate the variability of a score distribution and to see the extent to which this variability is associated with other factors.

In a design with one facet there are three factors that can influence variability.

1. The extent to which the targets differ.
2. The extent to which the items differ.
3. Measurement errors.

For these three factors there are different formulas to calculate the variance components (see table 13.3 in the book).

In a design with one facet, the ANOVA gives two main effects and a residue (error).

The result that we are most interested in is the target effect . This reflects the extent to which targets have different averages. The target effect is the 'signal' that a researcher is trying to discover. In a design with one facet, the residual effect is the 'noise' that potentially marks the signal of the target effect. If the measurement is good, participants who score high on one item will do the same on the other items. When the items are inconsistent it indicates that there are no clear differences between the individuals and that the items may not be good reflections from the construct.

2. D study

During this phase, the psychometric quality of different measurement strategies is estimated that can help in planning a good measurement strategy for the research in question. In this phase, coefficients or generalizability for different measurement strategies are estimated. These coefficients vary between 0 and 1.0. The following applies:

Generalizability coefficient = Signal / ( Signal + Noise )

For the formulas and one worked out to calculate the relative generalizability of the differences between targets, see book page 435-436. 

There is an important difference between a design with one facet and a design with multiple facets. This difference lies in the complexity of the components that influence the variability in the data. When a new facet is added, new components are also added. This complexity makes the ' noise ' or 'error element' of the generalizable coefficients more complex.

Examples of 'one-facet designs' and 'multiple facet designs' can be seen in the book.

Which other measurement designs are there? 

There are at least four important ways in which a G theory analysis can differ compared to another G analysis:

1. The number of facets;
2. random vs. fixed facets;
3. crossed vs. nested designs;
4. relative vs. absolute decisions.

1. The number of facets

The more facets there are, the larger and more complex the design and the more effects there are that generate variance components. The basic logic and process of the G theory is however the same with designs with fewer facets.

2. Random vs. fixed facets

If there is a random facet, then the items of the facet are chosen randomly from a sample of a universal number of items.

If there is a fixed facet, then all conditions of the facet are included in the analysis. People do not want to generalize outside the conditions used in the analysis .

The difference between using random or fixed facets can have important psychometric consequences. It can influence the psychometric quality of the research. It can also have consequences for the generalizability of the quality of the measurements.

3. Crossed vs. nested designs

In a multi-faceted analysis, the pairs of facets are crossed or nested. When a pair is crossed, all possible combinations of two facets are included in the analysis. If not all possible combinations are included, then it is a nested design. Determining this is important because it determines which effects can be estimated in a G analysis.

4. Relative vs. absolute decisions

A G theory can be used to make two types of decisions. Relative decisions contain the relative order of participants. When tests are used to make relative decisions, they are often referred to as norm- referenced tests . Absolute decisions are based on the absolute level of the score of an individual. When tests make such decisions, they are called criterion-referenced tests . Determining the difference between these two decisions is important because it influences the way noise or error is perceived. It influences the number of variance components that contribute to error when generalizibility coefficients are calculated. In most studies, researchers are more interested in the relative perspective than the absolute perspective. They are more interested in understanding relative differences between participants' scores on a measurement. So why some people score relatively high and others relatively low.

The Generalizability Theory (G theory) helps us to distinguish the effects of multiple facets and then to use different measurement strategies. It is an ideal framework for complex measurement strategies in which several facets influence the measurement quality. This is a fundamental difference compared to the classical test theory (CTT), where different facets are not assumed.