Predicting a criterium-score - a summary of an article by Oosterveld & Vorst (2010)

Critical thinking
Article: Oosterveld & Vorst, 2010
Voorspellen van een criteriumwaarde

Prediction-table: cross-table of criterium-values and test-scores

The test-scores and criterium-values can lay on a (almost) continuous scale, or have a dichotomous character.
Usually, criterium-values are established by judgements of experts.
Commonly, a criterium-value is valuable, it is true for the time being.

The test-score and criterium-value can be established simultaneously or with a short or long period in between.

• Prediction: first the test-score is established, then the criterium-score
• Postdiction: first is the criterium-score established, then the test-score

This has effect on the interpretation of the table
With a long time in between, prediction becomes less stable.

Usually, criterium-values are placed in the vertical axis and test-scores on the horizontal axis.

• if the test-score on a criterium-value is higher, the person has more of the trait

Not everyone uses this system

Indices for the quality of prediction

Base rate or prevalence: the percentage occurrence of the trait in the population.
With a low prevalence, finding the trait is difficult.
The use of a test must lead to a higher percentage well detected cases (hits) than the prevalence. Otherwise, using the test is useless.

• Prediction-error or classification-error: the percentage wrongfully submitted cases by the test.
  It is a global indicator of the performance of the test.
• Sensitivity or predictive accuracy: the percentage rightfully submitted cases that actually has the trait (hits).
• Specificity: the percentage of cases that is rightfully not submitted and that also doesn’t have the trait.

Sensitivity and specificity are direct clues to the predictive value of the test.

• Positive predictive value (PPV): the percentage that is rightfully detected with the trait by the test of the total persons that the test said has the trait.
• Negative predictive value (NPV): the percentage which the test rightfully said didn’t have the trait of the total of people the test said didn’t have the trait.

PPV and NPV are direct clues to the predictive value of the test.

Reliability of the prediction

The reliability of the prediction: the repeatability of the prediction on a certain point of time.

The reliability of the prediction can be established with cross-validation.

• the population is split a-select, and two prediction-tables are formed with both sub-populations.
  Differences in indices between tables give an indication of the prediction-reliability.

Stability of the prediction

Stability of the prediction: the repeatability of the prediction in the course of time.
Especially important when the predictions are about a period of time.

The stability of prediction can be established by doing a prediction of the criterium-values based on test-scores and changes in the prediction-tabel.
After some time, the prediction is checked by determining the criterium-value.

If all persons are admitted without selection, the stability of the whole prediction-table can be established.
If only the selected are researched, only the stability of the differences in hits and false-positives can be checked.

Use of tests in prediction

Utility of a test: the profit of the right number of classifications minus the costs of the number of wrong classifications minus the costs of the testing-procedure.
The utility is maximized if all cases are predicted rightfully.

• the prediction-error here is 0, and all other indices are 1.

You can change the cutoff-score

• a low cut-off score heightens the percentage (fals-)positives and lowers the percentage (fals-)negatives
• a high cut-off score lowers the percentage (fals-)positives and heightens the percentage (fals-)negatives

With each adjustment, the indices of values, the optimal use the the test, and the utility changes.