writing and evaluating test items

Question 1

how do you choose format of items?

Answer 1

Choice of format comes from objectives and purpose of the test

Question 2

Item writing guidelines

Answer 2

1. define clearly what you want to measure
2. generate and item pool
3. avoid long items (tedious to read)
4. keep reading difficulty appropriate (education level)
5. use clear and concise wording (avoid-double barreled and double negatives)
6. mix positively and negatively worded items in the same test
7. make sure items are culturally neutral as possible
8. make content relative to the purpose

Question 3

How to write MCQ items

Answer 3

vary position of correct answer

all distractors must be plausible

Question 4

True/false Qs

Answer 4

Both statements same length

Equal numbers of both

Question 5

5 types of item format

Answer 5

1. Dichotomous format
2. Polytomous Format
3. Likert format
4. category format
5. checklists and q sorts

Question 6

Dichotomous format

Answer 6

• 2 alternatives
• True/False
• Yes/No

Question 7

Dichotomous format advantages

Answer 7

1. ease of administration
2. quick scoring
3. requires absolute judgement

Question 8

Dichotomous format disadvantages

Answer 8

1. less reliable (50% of getting an item correct, less range of scores when it comes to analyses)
2. encourages memorisation
3. often truth is not black/white

Question 9

Polytomous format

Answer 9

more than 2 alternatives

MCQ questions

Question 10

Polytomous format- distractors

Answer 10

• incorrect alternatives
• ideal to have 3-4 distractors to retain pscyhometric properties
• must be as plausible as the correct answer
• no cute distractors
• make the test more reliable
• but difficult to find good distractors

Question 11

Polytomous format- advantages

Answer 11

• easy to administer and score
• requires absolute judgement
• less likely to guess correctly than a dichotomous test

Question 12

Correction for guessing

Answer 12

R – W/n – 1
Number of right answers minus the number of wrong answers divided by the number of choices for each item (minus 1)
R = number of correct
W = number of wrong
N = number of alternatives
Omitted answers are excluded in this calculation

Question 13

Likert Format

Answer 13

Named after Likert, who 1st used it for attitude scale

• idicates degree of agreement
• 6-point scale (or even number of options) used to avoid the neutral response
• Reverse score negatively-worded items
• use statments
• popular for attitude and personality scales

Question 14

Category Format

Answer 14

On a scale of 1 to 10…

Research suggests 7 best

Question 15

Category Format- disadvantages

Answer 15

1. Tendency to spread responses across all categories
2. Susceptible to the groupings of things being rated (context)
3. Element of randomness

Question 16

When is Category Format used?

Answer 16

1. People are highly involved with a subject
   E.g., asking people in townships to rate service delivery
   More motivated to make a finer discrimination
2. Want to measure the amount of something
   E.g., road rage experienced in a given situation
3. Make sure your endpoints are clearly defined

Visual analogue scale

Question 17

Checklists

Answer 17

• Common in personality measures

- A list of adjectives, check which ones describe you best

Question 18

Q-sorts

Answer 18

• place statements into piles
• piles indicate degree to which you think a statement describes a person/yourself
• category format implicit here

Question 19

Item analysis

Answer 19

Item analysis is a general term used to describe a set of methods used to evaluate test items. Item difficulty and item discriminability are the most basic of these methods.

Question 20

Item Difficulty

Answer 20

• the proportion of people who get a particular item correct
• higher value= easier the item
• p = number answered item correctly / number taking the measure

Question 21

Optimum difficulty level (ODL)

Answer 21

-between 0.30 and 0.70

Example: MCQ test with 4 alternatives
-4 answer options, therefore chance = 0.25
-Halfway between 100% and chance: (1.00 - 0.25)/2 = 0.375
-Add chance: 0.375 + 0.25 = 0.625
(Add chance because we require a difficulty level of at least chance)
-ODL = 0.625

Question 22

exceptions to optimum difficulty level

Answer 22

1. At times we need more difficult items e.g., selection process
2. At times we need more easier items e.g., special education
3. At times we need to consider other factors e.g., boost morale

Question 23

Item discriminability

Answer 23

Have those who did well on particular items also done well on the overall test?

Question 24

Good item discriminability when:

Answer 24

People who do well on test overall get the item correct (and vice versa)

Question 25

Discrimination Index (di)

Answer 25

Higher values indicate better discriminability

Question 26

Item discriminability- extreme groups method

Answer 26

-Calculated by looking at the number of people in the upper quartile who got the item correct divided by the number of people in the lower quartile who got the item correct -Essentially subtracting item difficulty between top and bottom 25% di = U/Nu – L/NL

Question 27

Item discriminability: The point-biserial method

Answer 27

-Also known as item-total correlation - Item correlations can also be used for Likert-type items, category format items, etc. Again, good items should be those that have a positive item-total correlation For example: If an item on a questionnaire measuring schizophrenia symptoms has a high correlation with total scores on the overall questionnaire, then the item is good at measuring schizophrenia symptoms Could use this correlation of an indicator to include or exclude from test/questionnaire in future Include higher and exclude lower

Question 28

Item characteristic curves (ICCs)

Answer 28

The relationship between performance on an item and performance on the overall tests tells us how well the item is tapping into what we want to measure. -A graphical display of item functioning Total test score plotted on X-axis Proportion (i.e., 0.23, 0.50, etc.) getting the item correct plotted on Y-axis -need discrete categories for scores.

Question 29

Item response theory (irt)

Answer 29

A different model of psychological testing -Makes extensive use of item analysis -Computer generates items Each of these items has a particular difficulty level -Computer gives you an item -If you answer it correctly, the next item will be of increased difficulty, if incorrectly, the next item will be of decreased difficulty -Looks at what you can do and only gives you what it thinks you can handle -Essentially, the test is ‘tailored’ to the individual Example: This person can answer most items correctly at the 0.30 (or 0.45 or 0.70, etc.) level of difficulty… Rather than: This person got 30% or 45% or 70% on this test.

Question 30

Test performance in irt- advantages

Answer 30

- Tests based on IRT can easily be adapted for computer administration - Quicker tests - Morale of test-taker is not broken down - Reduces chances of cheating

Question 31

Measurement precision: peaked conventional

Answer 31

- tests individuals at average ability best - doesn't assess high or low levels well - high precision for average ability levels, low precision at either end

Question 32

Measurement precision: rectangular conventional

Answer 32

- equal number of items assessing all ability levels | - relatively low precision across the board

Question 33

Measurement precision: adaptive

Answer 33

- tests focuses on the range that challenges each individual test taker - precision therefore high at every ability level

Question 34

Criterion-referenced tests

Answer 34

-Compares performance with some objectively defined criterion E.g., the extent to which performance on the QLT predicts success at stats in psychology Develop tests based on learning outcomes What is it that the student should be able to do? E.g., At the end of this lecture you should be able to: Describe an ICC Calculate item discriminability Calculate a point-biserial correlation

Question 35

Evaluating items in Criterion-referenced tests

Answer 35

- 2 groups: 1 given the learning 'unit'. 1 not given the learning 'unit' - collect scores; plot on graph and should form a V or U shape -bottom curve is the anitmode

Question 36

Limitations of Criterion-referenced tests

Answer 36

Tells you that you got something wrong, but not why Emphasis on ranking students rather than identifying gaps in knowledge ‘Teaching to the test’

Question 37

how does IRT differ to traditional testing methods?

Answer 37

it is defined by the level of difficulty of items answered correctly Instead of total test score as a traditional method does