Utility analysis - a summary of a part of The science of psychological measurement by Cohen

Critical thinking
Article: Cohen
Utility Analysis

What is a utility analysis?

Utility analysis: a family of techniques that entail a family of techniques that entail a cost-benefit analysis designed to yield information relevant to a decision about the usefulness and/or practical value of a tool of assessment.
It is not one specific technique used for one specific objective. It is an umbrella term covering various possible methods, each requiring various kinds of data to be inputted and yielding various kinds of output.

In a most general sense, a utility analysis may be undertaken for the purpose of evaluating whether the benefits of using a test outweigh the costs.

If undertaken to evaluate a test, the utility analysis will help make decisions whether:

• one test is preferable to another test for use for a specific purpose.
• one tool of assessment is preferable to another tool of assessment for a specific purpose
• the addition of one or more tests that are already in use is preferable for a specific purpose.
• no testing or assessment is preferable to any testing or assessment

If undertaken for the purpose of evaluating a training program or intervention, the utility analysis will help make decisions regarding whether:

• one training program is preferable to another training program
• one method of intervention is preferable to another method of intervention
• the addition or subtraction of elements t an existing training program improves the overall training program by making it more effective and efficient
• the addition or subtraction of elements to an existing method of intervention improves the overall intervention by making it more effective and efficient
• no training program is preferable to a given training program
• no intervention is preferable to a given intervention

The endpoint of a utility analysis is typically an educated decision about which of many possible courses of action is optimal.

How is a utility analysis conducted?

The specific objective of a utility analysis will dictate what sort of information will be required as well as the specific methods to be used.

Expectancy data

Some utility analyses will require little more than converting a scatterplot of test data to an expectancy table.
An expectancy table can provide an indication of the likelihood that a test-taker will score within some interval of scores on a criterion measure.

Taylor-Russell tables: increase the base rate of successful performance that is associated with a particular level of criterion-related validity.
The value assigned for the test’s validity: the computed validity coefficient.
But, the relationship must be linear.
Naylor-Shine tables: tells us the likely average increase in criterion performance as a result of using a particular test or intervention: also provides selection ratio needed to achieve a particular increase in criterion performance.

The expectancy table or chart: tells us the likelihood that individuals who score within a given range on the predictor will perform successfully on the criterion.

The Brogden-Cronbach-Gleser formula

Used to calculated the utility gain from the use of a particular selection instrument under specified conditions.
Utility gain: an estimate of the benefit of using a particular test or selection method.

Utility grain = (N)(T)(r_xy)(SD_y)(Z_in) – (N)(C)

N =the number of participants

T = tenure

r_xy = the criterion-related validity coefficient for the given predictor and criterion

Sd_y = the standard deviation of performance

Z_in = the mean score on the test for selected participants

C = the cost of the test for each applicant

Utility analysis: an illustration

Utility analysis has its own vocabulary.

Hit: a correct classification
Miss: an incorrect classification
Hit rate: the proportion of people that an assessment tool accurately identifies as possessing or exhibiting a particular trait, ability, behaviour, or attribute
Miss rate: the proportion of people that an assessment tool inaccurately identifies as possessing or exhibiting a particular trait, ability, behaviour, or attribute
False positive: a specific type of miss whereby an assessment tool falsely indicates that the test-taker possesses or exhibits a particular trait, ability, behaviour, or attribute
False negative: a specific type of miss whereby an assessment tool falsely indicates that the test-taker does not possess or exhibit a particular trait, ability, behaviour, or attribute

Base rate: the proportion of people in the population that possesses a particular trait, behaviour, characteristic, or attribute.

Return to investment: the ratio of benefits to costs.