Classical analysis of item scores - a summary of chapter 6 of A conceptual introduction to psychometrics by G, J., Mellenbergh

A conceptual introduction to psychometrics
Chapter 6
Classical analysis of item scores

The conventional way of scoring items is by assigning ordinal numbers to the response categories.
Usually, these item scores are ordered with respect to the attribute that the item is assumed to measure. But, these assignment of these ordinal numbers lacks a theoretical justification.

Usually, the analysis of test scores is supplemented by an analysis of the item scores.

Item score distributions

The scores of a given item have a distribution in a population of N persons.

• Location: the place of the scale where item scores are centered
• Dispersion: the scatter of the item scores
• Shape: the form of the distributions

Classical item difficulty and attractiveness

The location of the item score distribution is used to define the classical item difficulty (maximum performance tests) and classical item attractiveness (typical performance tests) concepts.

• Classical item difficulty: a parameter that indicates the location of the item score distribution in a population of persons.
• Classical item attractiveness: a parameter that indicates the location of the item score distribution in a population of persons.

The two definitions are the same.

Classical item difficulty and attractiveness are defined in a population of persons.
Population-dependent and may differ between populations.

The mean in mainly used for this.
The mean of a dichotomously scored item is called the item p-value.

Item score variance and standard deviation

The most common parameters that are used in classical item score analysis are the variance and the standard deviation of the item scores.

Items that have a small item score variance, have little effect on the test score variance.

The variance of dichotomous item scores is a function of the item p-value.
For a given sample size, the variance has its maximum value at p=.5.

Classical item discrimination

Location and dispersion parameters yield useful information on the items of a test.
But, these parameters do not indicate the extent to which an item contributes to the aim of a test to assess individual differences in the attribute that is measured by the test.

Classical item discrimination: a parameter that indicates the extent to which the item differentiates between the true test scores of a population of persons.
Defined in a population of persons, may vary between different populations.

The item-test and item-rest correlations

An appropriate index for discrimination between the true scores would be the product moment correlation between the item score and the true score in the population of persons.
Test taker j’s observed score is the estimator of his true score.
The population item-test correlation is estimated by the sample correlation.

Item-rest correlation: the product moment correlation between the item scores and the rest scores of the test, where the studied item is deleted.

The item reliability index

Item reliability index: the k^th item reliability index is the product of the item-test correlation and item score standard deviation of the k^th item in a population of persons.

Distractor analysis

A maximum performance item presents the test taker with a problem that has to be solved.
The response mode of this type of item is either the free(constructed) response or the choice (selected) response mode.
The most common response mode is multiple choice.

• A multiple choice itme has one correct option and a number of distractors. If the test taker selects the correct option his item score is 1, if he selects a distractor his item score is 0.
  Classical item difficulty and discrimination indices are used for this type of dichotomous scoring, but the test takers’ distractor choices contain information on the multiple choice item.

Item distractor popularity

Classical difficulty of a dichotomously scored multiple-choice item: the proportion of persons of the population who selected the correct answer to the item.

Item distractor popularity: the proportion of persons of a population who selected the distractor.

The item difficulty and the item distractor popularities are estimated in a sample of test takers by the proportions of test takes of the sample who selected the correct answer and the distractors, respectively.

The distractor popularities yield information on the appropriateness of the distractors. An unpopulair distractor is selected by only a small proportion of test takers.
Apparently, most test takers know that the distractor is an incorrect answer to the item, which means that the item can do without this distractor.

Item distractor-item correlations

The usual way of scoring multiple-choice items is by assigning a 1 to the choice of the correct option, and assigning a 0 to the choice of a distractor.

Item distractor-rest correlations: the product moment correlations of the separate dichotomous correct answer/distractor variables and the rest score.

The item distractor-rest correlations yield detailed information on item quality.
A positive distractor-rest correlation indicates that the distractor tends to attract test takers who have lower true scores than test takers who selected the correct answer.
A negative distractor-rest correlatoin indicates that the distractor discriminates in the wrong direction because the distractor tends to attract test takers who have higher true scores than test takers who selected the correct answers.

The internal structure of the test

The observed test score is the unweighted or weighted sum of the item scores.

The basic idea is that in a population of test takers the items that measure the same attribute are more highly associated among each other than associated with items that measure another attribute.
The association between items can be assessed by different coefficients.

• The conventional approach is to use a correlation coefficient to assess inter-item association. The internal structure of the test is studied by searching for clusters of items that are highly correlated within clusers and less correlated between clusters.

Analysis of inter-item product moment correlations

The conventional approach is to compute product moment correlations between the item scores, and to look for clusters of items that are highly correlated within the cluster and less correlated with items of other clusters.

The product moment correlation between the scores of two items in a population of persons is estimated by the sample product moment correlation.

Phi coefficient: the product moment correlation coefficient between two dichotomous variables.

The phi coefficient between two items can only reach a maximum of 1 if the p-values are equal.