Creating a Test

Question 1

Give examples of some of the issues to be considered when conceptualizing a test

Answer 1

Consider what’s been done before and if the proposed test offers anything new; who’s going to use it; context in which it will be used; how long or effortful the measure can be; any practicalities to be considered; any ethical issues to address

Question 2

Give an example of the sort of practicalities that might need to be considered when conceptualizing a test

Answer 2

Large enough font size if using older adults; Size of budget and amount of time; expense if administered to thousands of people, and skill of examiners

Question 3

What sort of ethical issues might you need to consider when conceptualizing a test?

Answer 3

Anonymity and whether this will create problems; whether there’s any potentially sensitive or offensive content

Question 4

What sort of issues should you consider when creating the materials for a new test?

Answer 4

Format of test items; scoring of test; whether items yield sufficient response variability (e.g. if norm-referenced); whether the items sample the domain of interest sufficiently; if items test people on the criterion; number of items (more items = more reliable but takes longer); whether you want/have face validity

Question 5

What key considerations might you need to take into account when drafting written content for a test?

Answer 5

Avoid ambiguity and lack of clarity; include reversed questions

Question 6

If someone asks you advice on designing a good multiple-choice question, what would you tell them?

Answer 6

Make sure distractors are plausible; reduce the chance of a respondent guessing the correct answer without actually knowing it

Question 7

What are selected-response questions?

Answer 7

They test recognition; e.g. multiple-choice questions; matching options; true/false

Question 8

What are constructed-response questions?

Answer 8

Where respondents generate the answer (recall); e.g. fill in the blank; write an essay/account

Question 9

In a multiple-choice question, what is the “stem”?

Answer 9

The question part

Question 10

Describe five strategies you could use for pilot research into your new test

Answer 10

1. Examine previous literature;
2. Interview people who might know something about the area;
3. Give people a more open-ended form of the test to generate ideas for items;
4. Get people to complete a preliminary or “faked-up” version of the test and comment on it;
5. Get people to give a running commentary about their thoughts as they complete a draft of the test

Question 11

List five strategies you could use for evaluating the quality of items in an achievement test

Answer 11

Item difficulty index; Reliability item discrimination index; Validity item discrimination index; If multiple choice, examine pattern of responses across options; Further in-depth examination

Question 12

What’s the item difficulty index?

Answer 12

The proportion of respondents who get the item correct

Question 13

What the validity item discrimination index?

Answer 13

The item’s contribution to the scale’s relationship to an external criterion measure

Question 14

When examining the quality of items, what further in-depth questions need to be asked?

Answer 14

Are there items where respondents expected to get it correct get it wrong?; Are there items where respondents expected to get it wrong are scoring above chance?

Question 15

Describe the four strategies you could use to evaluate item quality in a personality-type test

Answer 15

Examine histogram/frequency table of responses to each item to determine the spread; reliability item discrimination index; validity item discrimination index; further in-depth examination (expectations of high vs. low scorers)

Question 16

How do you calculate the item difficulty index?

Answer 16

Count the number of correct responses for each item and divide by the total number of people

Question 17

If you have a norm-referenced aptitude-type test, what is your goal, and what’s the optimal strategy to achieve this?

Answer 17

Goal is to maximize the opportunity for the test to tell apart people high or low on the trait or characteristic; adjust your items so the item difficulty index is 50% (p=.5) so half the respondents get it correct

Question 18

What is the optimal value for the item difficulty index for a six option multiple-choice question in a norm-referenced aptitude test?

Answer 18

100/6 = 16.67% + 100% = 116.67%/2 = 58.34%; so the optimal difficulty is halfway between chance and everyone being correct

Question 19

If the item difficulty index on an item is .92 on a norm-referenced test, what does this mean?

Answer 19

That 92% of people got it right and it was probably easy (ceiling effect); doesn’t tell people apart

Question 20

In a norm-referenced test, what if the item difficulty index is .11?

Answer 20

11% of people got it right which is less than chance; it could be really hard or there’s an error; floor effect

Question 21

How can you examine the spread of scores for items in a personality test?

Answer 21

Plot a histogram or output a frequency table for each item; check if the scores are spread across all response options for all items without too much skew

Question 22

If individual score items on a personality test are skewed, does this mean the total score will be?

Answer 22

Not necessarily

Question 23

What is the item discrimination index?

Answer 23

It determines which items contribute the most to the internal consistency of the test (reliability); and how each item contributes to the overall validity

Question 24

How do you calculate the item discrimination index with respect to internal consistency?

Answer 24

1. Count up correct answers for each person
2. Define upper and lower groups as top and bottom 25% based on their total score
3. For each item, count up the number of people in the high scoring group who got it correct (U) and the people in the low scoring group that got it correct (L)
4. Plug in the formula

Question 25

What is the maximum d can be on the item discrimination index, and what does it mean?

Answer 25

+1 - all high scorers get it correct and low scorers get it wrong; the closer d is to 1 the more similar it is to the other items and more reliable

Question 26

What does an item discrimination index of -1 mean?

Answer 26

It’s the minimum d can be - all low scorers get it correct and high scorers get it wrong (the opposite of what we want so maybe a scoring error)

Question 27

What if d=0 on the item discrimination index?

Answer 27

There’s no difference in performance between high and low scorers so no correlation with other items (measuring a different thing)

Question 28

How do you calculate the item discrimination index with respect to criterion validity?

Answer 28

In addition to the quiz, all respondents complete a previously validated test measuring the same thing (to obtain criterion score), then we examine the correlation between total quiz score and criterion score; define upper and lower groups as top and bottom 25% based on criterion score, and use the d formula

Question 29

Why do we examine the correlation between the total quiz score and a criterion score when calculating validity item discrimination index?

Answer 29

To see how each item contributes to the overall validity in case we want to increase validity by removing items

Question 30

What do the values of the item discrimination index mean if groups are defined by an external criterion?

Answer 30

The closer d is to 1, the more the item is contributing to the test’s criterion validity; if d = 0, then the item effectively has zero criterion validity (can’t predict the criterion at all)

Question 31

How do you calculate the item discrimination index for personality type tests?

Answer 31

Choose a threshold and label above or equal to as correct, and below as incorrect (halfway along the scale; e.g. If 1-5, threshold is 3); code responses and proceed with d calculation

Question 32

How could you go about evaluating items for a criterion-referenced test?

Answer 32

1. Test an experienced group who should fulfill the criterion;
2. Compare with a novice group who shouldn’t fulfill the criterion;
3. Work out the item discrimination indices for both groups (substituted for upper and lower group)
4. If multiple choice, see if novices perform above chance on any questions
5. Examine items that experienced people are scoring poorly on

Question 33

What can examining the pattern of responses across items for high and low scorers in a multiple-choice test tell you?

Answer 33

It can tell how the item is working and look at the spread across distractors: if one isn’t chosen at all then it’s redundant; if a distractor is chosen more than the correct answer, then the wording may be misleading or it’s actually correct and there’s an error

Question 34

Which of the item analyses covered in this lecture are appropriate for speed tests, and why?

Answer 34

None of them, as what discriminates between people is not how many they get correct but how many they complete; it doesn’t focus on the content of the item