The features and quality standards of diagnostic instruments - Verhoeven - 2014 - Article

What are the most common diagnostic instruments?

Psychologists make use of three different kinds of methods and instruments which are scientifically demonstrated. These are interview techniques, psychological tests, and observation methods. These three instruments are called the ‘golden trio of testing’.

In practice, psychologists often use a combination of these three diagnostic instrument types. These instruments also come in a variety of forms. For example, interviews can be very structured to completely free and open. Observations can also be made in different ways, again ranging from a rigid structure to no structure at all.

One goal of psychology is to make vague, subjective, or intangible phenomena objective, precise, and measurable. They use these instruments to do so. However, the instruments used in psychology are often less accurate compared to other exact sciences. No interview, psychological test, or observational technique will lead to a perfect prediction in every single case. For some people, this is enough to discard psychological measurement instruments. However, research has shown that in almost all cases, the quality of personal assessments is lower compared to that of a reliable instrument.

When is a test a good test?

There are a lot of tests and questionnaires on the internet which are often stated to be ‘psychological tests’. However, there are some important differences between these tests and official psychological tests. These are the key differences:

Professional tests are based on scientific research, such as a scientific personality theory or a statistical method.

• The scales used are homogeneous and pure.
• The test itself has been tested for reliability and validity.
• The method for administering the test is described in precise terms, and is generally highly standardised.
• Candidates’ test scores can be compared to sufficiently large and representative norm groups.
• Research has been conducted on the test’s validity, so on whether it actually measures what it claims to measure.

Applied psychologists are often test users. Test constructors are research psychologists. So, applied psychologists do not need to know all the details of scientific and statistical methods. However, they do need to be aware of the general terms.

What is the scientific underpinning of diagnostic instruments?

The authors state that all people are actually psychologists: everyone has an idea about how the human mind works, and why we do things. So, what is the difference?

Psychologists regard themselves as a member of a scientific community. The ideas and opinions that they have, are seen as temporary hypotheses that are subject to discussion and empirical research in academic fields. Also, they do not come up with ideas randomly. Instead, they often use other, empirically proven, psychologists’ theories. What this means is that many ideas that were once regarded as valid, are now outdated. An example of this is Freud’s ideas. So, a scientific mindset involves accepting that science is a never-ending discussion in which ideas are accepted or rejected based on hard evidence.

The use of pure, homogeneous constructs

Factor analysis is a statistical method that aims to reduce data into distinguishable constructs or factors. So, it looks at patterns and correlations within large and complex datasets.

Reliability

Reliability is an important aspect of psychological tests. In other words, it is a key psychometric concept. Reliability refers to the internal properties of a test or measurement method. It relates to the absence of flaws that can impede or distort the measurement process. For example, we have a set of scales, which measure someone’s weight. Some factors might lead to a distorted outcome, for example:

• The person reading the weight is careless or cannot see very well and writes down the wrong number;
• The spring in the scale is rusty;
• The person being weighed keeps their clothes on, but the procedure states that they should wear only underwear;
• The person being weighed secretly leans on a table, which produces a lower reading;
• The person being weighed has been suffering from serious diarrhea for several days, and so the reading is lower than it should be.
• In psychological tests, these factors may distort the outcome:
• A person may score lower on an intelligence test due to a lack of motivation;
• Adolescents don’t take their learning test style seriously, and provide random answers;
• A job applicant from another culture is not sufficient enough in Dutch, and does not understand certain sayings or expressions in a test.

So, reliability refers to the absence of ‘noise’ in the data. Recently there have been statistical methods developed that deal with these kinds of issues. These methods do not determine where the noise is coming from, instead, they look at how big the effect of the noise is on the outcome.

Four ways to measure reliability

There are four ways to determine the reliability of an instrument. The first way refers to homogeneity and purity. This means that when there is a scale (such as extraversion), all the items in the scale really do measure extraversion. So, when someone scores high on question 1, he or she is expected to also score high on question 2. So, all items on the scale have a high mutual correlation.

Another method is the ‘split-half’ method. This divides the items into two random groups and then measures the correlations between the two groups.

The third method is the test-retest method. This means that the same test is administered to the same group of people, with a certain period in between. If this test is supposed to measure a stable personality trait (such as intelligence), then the two scores on the test should be very similar. So, again, there should be a high correlation between the two sets of test results.

The fourth method is the interrater reliability. Sometimes, the same behavior can be interpreted in different ways. Therefore, we can use two different raters and then compare their ratings. Again, there should be a high correlation between the two different ratings.

The criteria for reliability

Validity is the most important aspect of a test, but a test that is not reliable can never be valid. The literature sets minimum reliability of 0.8 for personality and intelligence tests that are used to make decisions about people’s lives. When tests are used more for recommendations, the minimum reliability is 0.7. When tests and instruments are used in experiments, the minimum reliability is 0.6.

What is validity?

Validity is defined as ‘the extent to which a test achieves its purpose’. So, is the test a good predictor of future behavior? So, does a CITO test match students’ performance later in secondary school?

There are different types of validity, with the most important being:

• Construct or content validity
• Predictive or criterion validity
• Face validity
• Construct or content validity

Construct validity is about whether the test adequately defines the concept that the researchers want to measure.

Predictive, external or criterion validity

This is the most important form of validity. This refers to a test’s ability to predict what will happen in the future, or to provide information on a phenomenon that is occurring now but which we cannot yet perceive. For the first, think about CITO tests which adolescents take in their last year of primary school. Based on this test, the adolescents are given advice for the level of education that they should follow in secondary school. So, what can be looked at is whether this test is a good predictor of performance in secondary school. For the latter, think of the use of questionnaires in medicine. Questionnaires about whether a patients has cancer makes claims about a circumstance that is already at hand, but that cannot be established with certainty. Only during surgery it can be clear whether this person has cancer or not. So, the questionnaire kind of makes a prediction.

Face validity

 This form of validity refers to the extent to which candidates or clients accept or understand the value of the method. Often, experts are asked about their opinion.

What is the COTAN?

The Dutch Association of Psychologists (Nederlands Instituut van Psychologen, NIP) formalized the features stated above in an official document, the COTAN. When someone develops a test, they can submit this new test to COTAN for assessment. Then, COTAN evaluates the test based on seven criteria:

1. Underlying principles and intended purpose
2. Quality of testing materials
3. Quality of the manual
4. Quality of the norms and norm groups (size, diversity, specificity)
5. Test reliability
6. Construct validity
7. Criterion validity

What makes a good diagnostician?

The quality of diagnostic research also depends on the quality of thought processes, expertise, and professionalism of the diagnosticians themselves. There are three different terms that are important for this, namely the fundamental attribution error, base rates and prevalence, and the professionalism of the diagnostician.

The fundamental attribution error

The fundamental attribution error refers to that laypeople and psychologists tend to overestimate the influence of personality attributes and underestimate situational forces. For example, when someone steals something, we often attribute this behavior to a person’s personality, instead of situational factors.

Base rates and prevalence

Base rates refer to the prevalence of certain disorders, attributes, and conditions. Applied psychologists need to know these base rates or prevalences before drawing conclusions. Otherwise, they might draw false conclusions about reality.

So, what makes a good psychodiagnostician according to the authors?

The qualities of a psychodiagnostician can be reduced to the following seven core competencies:

1. Diagnosticians must be able to take a complex or unclear problem from a client or costumer, and clarify it so that it is transparent both to themselves and to others.
2. Diagnosticians must be able to take complex, personal accounts from clients or customers that use personal and subjective terms, feelings, images and metaphors, and translate them into objective, verifiable concepts. Applied psychologists must be masters in the operationalisation of concepts that the outside world generally considers to be intangible or ephemeral. They must be able to make the subjective objective and verifiable, the immeasurable measurable, and express the qualitative in quantitative terms without sacrificing meaning or nuance
3. Diagnosticians require a knowledge of the various contexts, roles, demands or environmental conditions that clients/customers face (or may face). In cases where these factors are implicit or unclear, diagnosticians must be able to clarify and communicate them. For example: if a student goes to an applied psychologist to ask whether they more suited to vocational or academic study, the psychologist must be (or become) aware of the specific differences between vocational and academic programmes, and be able to define the personality traits that determine success and failure in both contexts. Only with this knowledge can the psychologist translate the diagnostic results into an appropriate recommendation
4. Diagnosticians must be familiar with the range of available instruments (tests, interview techniques and observation methods) so that they can select or develop appropriate tools to actually measure the defined constructs. The accompanying problem analysis must also be thorough and specific enough, and the psychologist's knowledge of the instruments must be broad enough to enable them to formulate concrete hypotheses and respond to the future results of the measurements.   They use the hypothesis testing model (HTM) to render ideas and presuppositions explicit, and are prepared to subject their ideas and presuppositions to critical discussion with themselves, with their peers and with the know‐how incorporated into the instruments
5. Once the results are available, diagnosticians use them to verify the accuracy of previous hypotheses. If the hypotheses are falsified, diagnosticians can use the existing/supplementary data obtained to identify the holes in their previous analysis and opinion‐forming process, and are prepared to go through the HTM cycle once more.    Diagnosticians must also be able to escape ruts in their thinking, and approach the needs of clients and customers in new, open and creative ways. They must not be held back by their egos, but instead, be prepared to accept former incorrect assessments as such, and see them as necessary steps in the investigative process in order to solve client/customer questions and problems. In addition, they must be able to spot patterns in the test results obtained, thus reaching a deeper understanding of the question/problem and of potential solutions for the client or customer.
6. Diagnosticians need to be able to express themselves effectively verbally and in writing and to explain their research findings to customers and clients. Client reports and/or verbal recommendations must be clear enough so that the client or customer can understand them and gain a deeper understanding of their situation, needs or problem so that they accept the proposed interventions and energy is mobilized to initiate the change process.
7. Lastly, diagnosticians must not only be skilled in the application of their range of instruments, but they must also possess the communication, social and political skills necessary to generate and maintain acceptance not only from the client or customer but also from various other stakeholders.   They must realize that in practicing their profession they exert significant influence on important decisions. They must effectively negotiate and navigate the political landscape that inevitably arises, and always act in accordance with high professional and personal standards. They must also be prepared to be assessed on this basis.