What is validity? - Chapter 8

What is validity?

For more than 60 years the following basic definition of validity was assumed: Validity is the extent to which a test measures what it is intended to measure. Although this definition has been used very much and still is, the concept of validity is presented a little too simple with this definition. A better definition would be that validity is the extent to which the interpretation of test scores for a specific purpose is supported by evidence and theory.   

This more advanced definition leads to three important implications:

1. Validity refers to the interpretation of test scores regarding a specific psychological construct, it is not about the test itself. This means that a measurement is not valid or invalid, but that validity relates to the interpretation and use of measurements.
2. Validity is a matter of degree, it is not an "all or nothing" construct.
3. Validity is entirely based on empirical evidence and theory.   

Why is validity important?

Validity is perhaps the most important point regarding the psychometric quality of a test. This point is further substantiated in this section based on the role and importance of validity in psychological research and practice.  

Validity is needed to be able to perform psychological research. The goals of scientific research (describing, predicting, explaining a determined aspect in the world) depend to a large extent on our ability to manipulate and measure specific variables. 

Suppose we are interested in the relationship between violent games and aggressive behavior. The hypothesis is that children show more aggressive behavior when they play a lot of violent games. In order to test this hypothesis, it is important to measure the variable "aggressive behavior". Only then can we state with a higher (reasonable) degree of certainty that violent games are indeed associated with increased aggressiveness. If we do not measure aggressive behavior, this conclusion must be seriously questioned. So without test validity we cannot make reliable statements about the role of video games on aggressiveness. 

More generally, we can state that without test validity, test-based decisions about individuals can be wrong and sometimes even harmful.  

What is the current perspective on validity and which facets play a role in this? 

For years there was a traditional perspective on validity, in which there are three types of validity were identified:

1. Content validity.
2. Criterion validity.
3. Construct validity.

Nowadays construct validity is seen as the essential concept in validity. Construct validity is the extent to which a test score can be interpreted as a reflection of a certain psychological construct.     

Three major organizations (AERA, APA, and NCME) published a revision of Standards for Educational and Psychological Testing in 2014, emphasizing five facets of construct validity. Construct validity is determined by five types of information:

1. Content;
2. internal structure;
3. response process;
4. associations;
5. consequences.    

1. Content

First, the (construct) validity is determined by the extent to which the actual content of a test is related to what should be included in the test. In other words, when a test aims to measure intelligence, the items of the test must reflect the important facets of intelligence. This is also referred to as content validity . Content validity is usually evaluated by experts who are familiar with the relevant construct. There are two threats to content validity: (1) constuct-irrelevant items; (2) construct underrepresentation . A test must not contain any content (items, questions) that are not relevant to the construct that the test is intended to measure. Construct irrelevant content is outside the main construct that the test is intended to measure, and it lowers validity. Second, a test must contain content that covers the full range of the construct. Again, if this is not the case, then the validity is reduced. In short, it means that a test must cover the full range of the construct, no more and no less.        

A closely follow this up concept is impressed validity (face validity). Impression validity is the extent to which a measurement appears to be related to a specific construct according to nonexperts (such as respondents or legal representative).     

2. Internal structure

A second important point for the validity of test interpretations is the internal structure or dimensionality of a test. The internal structure or dimensionality of a test is the way in which parts of the test are related to each other. Here too, an important component is: the extent to which the intended structure overlaps with the structure that it should have. The internal structure of a test vaa assessed by factor analysis (discussed earlier in Chapter 4 and is discussed further in Chapter 12). Some items from a test correlate more strongly with certain items than others. A set of items that strongly correlates with each other and forms a cluster of items is also called a dimension or factor. Factor analysis is used to identify the presence and nature of those factors. In addition, factor analysis is also used to map the associations between factors within a multidimensional test. Finally, factor analysis is used to identify which items are linked to which factors.        

3. Response process

A third type of proof of validity is the correspondence between the psychological processes that respondents actually use when they complete a measurement and the processes they should use. Many psychological measurements are based on assumptions about the psychological processes that people use when they complete a certain measurement. For example, for an item such as "I often go to parties," the researcher logically assumes that respondents try to remember how often to go to parties and then assess whether that number can be seen as "a lot." Various procedures have been developed to assess the response process validity. Some of these procedures use direct evidence (for example, by simply interviewing the respondents and asking them what considerations they had when completing the measurement), others using indirect evidence (for example, by eye tracking) .      

4. Associations

A fourth type of proof of validity is the extent to which test scores are associated with other variables. This is about the degree of similarity between the actual associations with other measurements and the associations that the test should have with other measurements.    

There are different types of evidence. Convergent evidence refers to the extent to which test scores correlate with other measurements from relevant constructs (ie, the extent to which test scores correlate with measurements with which they should correlate). Discriminant evidence is the extent to which test scores are uncorrelated with testing of uncorrelated constucts (ie, the extent to which test scores do not correlate with measurements with which they should not correlate). In addition, a distinction can be made between concurrent and predictive validity. Competitive validity refers to the extent to which test scores correlate with other relevant variables that were measured at the same time. Predictive validity refers to the extent to which test scores correlate with other relevant measurements that were measured at a later time.       

5. Consequences

The fifth and final source of information for validity is formed by the consequences of testing. Are the test scores equally valid for men and women? Are the consequences of the test the same for different groups? Is there beooge of unintended differential effects of the test on certain groups?

What other perspectives on validity are there?

Not everyone agrees with the validity perspective that we discussed in this chapter. Many other perspectives on validity are possible, but a detailed description of these perspectives goes beyond the purpose of this book. Three other perspectives for assessing validity are, however, briefly explained. 

1. Criterion validity places less emphasis on the conceptual meaning or interpretations of the test scores. Test users sometimes just want to get groups separated by means of a test and then they do not find it important which construct is behind it. Criterion validity is the extent to which the test scores can predict the criterion variables. This also includes competitive validity and predictive validity. The psychological significance of test scores is relatively unimportant, because the only important thing is to separate the groups. It is nowadays considered that criterion validity has become part of construct validity. 
2. Another alternative is to learn what the test scores actually mean instead of testing certain theoretical hypotheses about the test scores. Researchers can also look at the real meaning of the test scores and draw up an evaluation. This is called the inductive approach to validity . The purpose of the inductive approach is to find out the full meaning of the test scores. This means that the construct can also be changed later. In applied research, for example, it can be used to get a specific job performance test . In the research field it is applied to discover new areas and to develop a theoretical basis for this. Test developers generally do not spend much time and effort on further developing existing studies.    
3. One can also look at validity by emphasizing the connection between tests and psychological constructs. A test is only a valid measurement of the construct if the construct influences the performance of the test subjects during the test. According to Borsboom, Mellenbergh and van Heerden (2004), the first objective of validity is to provide the theoretical explanation for the outcome of the measurement.

In which respects do reliability and validity differ?

It is important to know the difference between reliability and validity. Reliability is the extent to which the differences in the test scores between people correspond to the real differences in the trait, whatever trait it may be. We can discuss the reliability of the test without interpreting the test scores. Reliability is used to look at the reactions to the test and validity is used to look at the interpretation of the test scores. Validity is also more connected to psychological theories, while reliability is more a matter of quantity. Conceptually, there is often no validity without reliability, but reliability is possible without validity. If a test is reliable, that does not mean that it is valid. Tests can be reliable and not valid.

For more than 60 years, the following basic definition of validity was assumed: Validity is the extent to which a test measures what it is intended to measure. Although this definition has been used very much and still is, the concept of validity is presented a little too simply through this definition. A better definition would be that validity is the extent to which the interpretation of test scores for a specific purpose is supported by evidence and theory.