A conceptual introduction to psychometrics, development, analysis, and application of psychological and educational tests, by G. J. Mellenberg (first edition) – Summary chapter 2

Test development consists of several steps:

1. The construct of interest
   The latent variable (construct) that the test is supposed to measure has to be specified. Latent variables can vary in scope (1), content (2) and between educational and psychological variables (3).
2. The measurement mode
   The test has to specify how the latent variable will be measured. This includes three modes: self-performance mode (1), self-evaluation mode (2) and other-evaluation mode (3). Modes can be reactive or nonreactive. In a reactive mode, test takers can deliberately distort their construct value. In a non-reactive mode, test takers cannot deliberately distort their construct value.
3. The objectives of the test
   The test has to specify the objectives of the test. Tests can be used for practical and scientific purposes and objectives can be at the level of an individual test taker or at the level of a group of test takers. Objectives distinguish between description, diagnosis and decision making.
4. The population
   The target population, the set of persons to whom the test has to be applied has to be specified. This often includes inclusion and exclusion criteria.
5. The conceptual framework
   This is the theoretical framework on which the test is based. It is used to make conceptual distinctions and organize ideas.
6. The item response mode
   The test has to specify how test takers respond to the items. This can be a free- or constructed response mode, which includes short-answer items and essay items. It can also be a choice or selected response mode. It can make use of frequency or intensity response scales and of endorsement response scales.
7. The administration mode
   The test has to specify how the test is administered. Administration mode can be oral (1), paper and pencil (2), computerized (3) and computerized adaptive test administration (4). Computerized adaptive test administration adjusts the difficulty of the test according to the level of the test taker.

Response scales can be dichotomous (two ordered categories), partial ordinal-polytomous (more than two ordered categories) and ordinal-polytomous (completely ordered categories). There are several item-writing guidelines:

1. Focus on one relevant aspect
   This means that items should not test two aspects at the same time.
2. Use independent item content
   Items should be independent of each other, but this is not necessary if the questions are based on a reading passage.
3. Avoid overly specific and overly general content
   Overly specific and overly general content lead to ambiguity in the answers.
4. Avoid items that deliberately deceive test takers
   Items that distract test takers’ attention from the problem that they have to solve should be avoided.
5. Keep vocabulary simple for the population of test takers
   For native speakers, the items should not require the reading skill beyond that of a twelve year old.
6. Put item options vertically
7. Minimize reading time and avoid unnecessary information
   Unnecessary information obscures the content and distracts test takers from the problem that they are trying to solve.
8. Use correct language
9. Use non-sensitive language
   An item should not foster stereotypes, should not contain ethnocentric or gender-based underlying assumptions, should not be offensive, should not contain controversial material and should not be elitists or ethnocentric.
10. Use a clear stem and include the central idea in the stem
11. Word items positively and avoid negatives
12. Write three options unless it is easy to write plausible distractors
13. Use one option that is unambiguously the correct or best answer
14. Place the items in alphabetical, logical or numerical order
15. Vary the location of the correct option across the test
16. Keep the options homogeneous in length, content and grammar
    Item-writers tend to make the correct option longer than the distractors.
17. Avoid ‘all-of-the-above’ as the last option
18. Make distractors plausible
    Distractors should all seem plausible to test takers who do not know the correct answer.
19. Avoid giving clues to the correct answer

The responses to free- or constructed response items have to be rated by raters. There are several item-rating guidelines:

1. Rate responses anonymously
2. Rate the responses to one item at a time
   All the responses to one item should be rated before moving on to the next one
3. Provide the rater with a frame of reference
   Raters should be given instructions, schemes or ideal responses that they can use as a frame of reference.
4. Separate irrelevant aspects from the relevant performance
   Only concentrate on relevant performance (e.g: don’t focus on writing in a maths test).
5. Use more than one rater
6. Re-rate the free responses
   The same free responses should be rated on more than one occasion.
7. Rate all responses to an item on the same occasion
8. Rearrange the order of responses
9. Read a sample of responses
   A sample of responses should be read before the start of the rating.

Pilot studies are conducted to test the quality of the concept items. Items are reviewed on content, technical aspects and sensitivity. Test takers are interviewed on their thinking while working on an item. Concurrent interviews include thinking aloud while working on the item and retrospective interviews include recollecting the thinking after an item.

Coefficient Kappa gives the consistency between different occasions for a rater and uses the following formula:

Kappa= O-E1-E(E<1)

E refers to the expected proportion of identical ratings and O refers to the observed proportion of identical ratings. E can be calculated by multiplying the marginal proportions.

E can be calculated by multiplying .69 with .71 + 12 * .10 and so on. O can be calculated by adding the diagonal cells with each other (.68+.09+.18).