A conceptual introduction to psychometrics, development, analysis, and application of psychological and educational tests, by G. J. Mellenberg (first edition) – Summary chapter 6

An item score distribution can be described by location (1), dispersion (2) and shape (3). Item difficulty is a parameter in maximum performance tests. More test takers fail on more difficult items. Item attractiveness is a parameter in typical performance tests. More test takers choose attractive items. The item difficulty / item attractiveness is equal to the item mean.

Items with small variances do not contribute much to the overall variance. There is a danger of small variances due to floor / ceiling effects in Likert scales (e.g. items with low attractiveness). Large item correlations result in high reliability. Item discrimination refers to how well a given item can distinguish between people that differ on the underlying construct.

The item-test correlation is the correlation between the scores on a given item and the test scores. Items that discriminate well have a high item-test correlation. It uses the following formula:

It is the sum of item score k for test taker j minus the mean for item k times the test score for test taker j minus the mean of the test score divided by the square root of the sum of the item score k for test taker j minus the item mean for item k squared times the sum of the test score for test taker j minus the mean of the test score squared. In other words, it is the covariance between item k and the test score divided by the standard deviation of item k times the standard deviation of the test score.

The item-rest correlation is the correlation between the scores on a given item and the rest score, the score without that item. It is used because in the item-test correlation, correlation is biased upwards as you are correlating an item with itself. It uses the following formula:

It is the sum of the item score k for test taker j minus the mean for item k times the test score for test taker j without item k minus the mean of the test score without item k divided by the square root of the sum of the item score k minus for test taker j minus the mean for item k squared times the sum of the test score for test taker j without item k minus the mean of the test score without item k squared. In other words, it is the covariance between item k and the test score without that item (rest score) divided by the standard deviation of item k times the standard deviation of the test score without item k (rest score standard deviation).

The item-reliability index uses the following formula:

It is the correlation between item k and the test score times the standard deviation of item k. It uses the item-test correlation and not the item-rest correlation.

The classical difficulty of a dichotomously scored multiple-choice item is the proportion of persons in a population who selected the correct answer to the item. The popularity of a distractor is the proportion of persons in a population who selected the distractor. The item distractor-rest correlation is the correlation between the correct answer / distractor variables and the rest score. A positive distractor-rest correlation indicates that that the distractor tends to attract test takers who have lower true scores than the test takers who selected the correct answer. A negative distractor-rest correlation indicates that the distractor tends to attract test takers who have higher true scores than the test takers who selected the correct answer. A positive distractor-rest correlation is desirable.