Study Guide: Test Construction

Question 1

6 Steps of Test Construction

Answer 1

• Define Test’s Purpose
• Preliminary Design Issues
• Item Preparation
• Item Analysis
• Standardization and Ancillary Research
• Preparation of Final Materials and Publication

Question 2

Test Purpose

Answer 2

• What will be measured?

* Who is the target audience and does the construct match the group?

Question 3

Preliminary Design Issues definition

Answer 3

Anything that introduces error. Must strike a balance between efficiency and accuracy as well as meet breadth and depth

Question 4

Examples of Preliminary Design Issues

Answer 4

• Mode of administration
• Length–longer is more reliable (up to ~15min)
• Item format (T/F, multiple choice, essay)
• # of scores (For example, “Depression is multi-faceted and has many scales)
• Training
• Background research

Question 5

What is the most important Preliminary Design Issue?

Answer 5

Background research!

Question 6

4 parts of Item Preparation

Answer 6

• Stimulus
• Response
• Conditions governing responses
• Scoring procedures

Question 7

Stimulus

Answer 7

The question itself is a stimulus.

*We are trying to provoke a specific response correlated with the construct driven ONLY by the stimulus

Question 8

Response

Answer 8

The behavior you are looking for that is correlated to a construct

Question 9

Conditions governing responses

Answer 9

What are your rules? Is there a time limit? Are they able to ask questions?

Question 10

Scoring procedures

Answer 10

Formula or rubric used to formulate final scores

*Make sure each facet is represented (weighted)

Question 11

Types of Test Items

Answer 11

• Selected-Response Items

* Constructed-Response Items

Question 12

Selected-Response Items

Answer 12

Where you know all possible responses without bias

*T/F, multiple choice, Likert scale, etc.

Question 13

Constructed-Response Items

Answer 13

Responses are unknown/more nebulous

*Essays, oral responses, performance assessment

Question 14

Benefits of Selected-Response Items

Answer 14

• One clear answer

* scoring reliability and efficiency

Question 15

Benefits of Constructed-Response Items

Answer 15

• No agreed-upon answer
• Bx can give further context
• Goes deeper into construct

Question 16

Item Analysis

Answer 16

• Item Tryout
• Statistical Analysis
• Item Selection

Question 17

Item Tryout

Answer 17

aka Pilot Test

• Get subjects similar to target population–cannot be same people used in actual survey
• 2-3x the items you think you will need.

Question 18

Statistical Analysis

Answer 18

• Difficulty
• Discrimination
• Distractor Analysis

Question 19

Item Difficulty

Answer 19

% of subjects taking the test who answered correctly

Question 20

Difficulty formula

Answer 20

p = # people correct // total

Question 21

What shows good variability for Difficulty?

Answer 21

.5

Question 22

Difficulty considerations

Answer 22

• Behavioral measure
• Characteristic of the item and the sample
• Extreme p values restrict variability
• More comparative than a ‘cut-off’

Question 23

Why is Difficulty a behavioral measure

Answer 23

It taps into individual differences in holding the construct

Question 24

Item Discrimination

Answer 24

• Assumption that a single item and the test measure the same thing–comparing items to other items within the test
• Looks at how well any single item is good at discerning who does/does not have a trait
• You want a high rate!

Question 25

2 Indices of Discrimination

Answer 25

* Index D | * Discrimination Coefficients

Question 26

Index D(iscrimination) formula

Answer 26

* Score and rank * Take top and bottom 27% D= (# correct upper - # correct lower) // # people in larger group

Question 27

Why do we generally focus on just the high/low 27%?

Answer 27

Look up

Question 28

Ranges of D(iscrimination)

Answer 28

.40 and up = good .30 to .39 = okay .20 to .29 = marginal .19 and below = poor

Question 29

Distractor Guidelines

Answer 29

* Plausible * Parallel in structure and grammar * Keep everything short * Mutually exclusive * Alternate placement * Limit 'all of the above' stuff

Question 30

D values for a distractor

Answer 30

* You want low, preferably negative (this means more in the low group chose it) * Zero: it might not be an equally plausible answer * Be cautious of large D values as well

Question 31

Why do we want consistency between distractors?

Answer 31

Moving away from randomness helps determine true measure of construct.

Question 32

Standardization and Ancillary Research

Answer 32

* Norming * Reliability Studies * Equating Programs

Question 33

Test Norming

Answer 33

Two steps: * Define target population * Select sample

Question 34

Sampling Methods

Answer 34

*Probability and Non-Probability

Question 35

Probability Sampling Methods

Answer 35

Every member of Population has a known non-zero chance of being selected * Random * Systematic * Stratefied

Question 36

Non-Probability Sampling Methods

Answer 36

Not every member of population has equal chance of being selected...and some have a zero chance of being selected * Convenience Sampling * Judgement * Quota * Snowball

Question 37

What is more important: original conceptualization or the technical/statistical work?

Answer 37

Original concept!

Question 38

What should you be thinking about, even at the original design stage?

Answer 38

Final Score Reports!

Question 39

Does the norming group need to be large?

Answer 39

Not if it is properly selected!