Chapter 8

Question 1

What’s an ideal item on a test for a norm-referenced test?

What about criterion?

Answer 1

Top scorers should get it correct, while low scorers should get it wrong.

The above doesn’t matter for criterion-based. An ideal test item is based on how well it assesses mastery.

Question 2

Scaling definition

Answer 2

Process of settings rules for assigning numbers in measurement.

Question 3

Stanine scale

Answer 3

Raw scores transformed into scores ranged from 1 to 9.

Question 4

Rating scale

Answer 4

Records judgements of oneself, others, experiences, or objects

Question 5

Summative scale

Answer 5

Final test score is a sum of all items

Question 6

Method of paired comparisons

Answer 6

Asked to choose an option based out of two options.

Question 7

Comparative Scaling

Answer 7

Sort options in comparison based on judgements. (eg. rank cards)

Question 8

Categorical scaling

Answer 8

Sort objects into categories (eg. sorting cards to “justified” “sometimes justified” “always justified”

Question 9

Guttman scale

Answer 9

Weaker to stronger expressions.

Agree with stronger ones will also agree with milder

Question 10

Direct vs Indirect estimation

Answer 10

Direct (like equal-appearing intervals) transforms responses to another scale.

Indirect is no need to transform to another scale.

Question 11

Selected-response vs. Constructed-response formats

Answer 11

Item formats. One is multiple options choose one, other is generate own answer.

Question 12

3 types of selected-response item formats

Answer 12

MCs, matching, t/f

Question 13

What are the names of the two columns in matching

Answer 13

Premises and responses

Question 14

Completion item

Answer 14

Fill in the blank item

Question 15

Computerized adaptive testing

What are the advantages of CAT?

Answer 15

Items are based on performance on previous items.

They reduce number of items needed and reduce measurement error (both by around 50%)

Question 16

Floor vs. Ceiling effects

Answer 16

Floor: assessment tool is bad at distinguishing testtakers at the low end of what’s measured. (all too hard)

Ceiling: everything’s too easy.

Question 17

Item Branching

Answer 17

Ability to customize content and order on the basis of previous responses.

Question 18

Class Scoring or Category Scoring

Answer 18

Responders gets placed in a class or category with other responders based on their responses.

Question 19

Ipsative scoring

what conclusions can be drawn?

Answer 19

Compare score on one scale to another scale within a same test.

Only appropriate for intraindividual comparison, not interindividual

Question 20

What makes a good test item?

Answer 20

Can discriminate testtakers.

All high scorers getting a particular item wrong is bad sign. Same for opposite (low scorers and getting that item right).

Question 21

Item analysis

Answer 21

Statistical procedures to analyze and identify good items for a test.

Question 22

4 possible analyses for test items

Answer 22

Difficulty, Reliability, Validity, and item discrimination

Question 23

How to calculate index of item’s difficulty?

item-difficulty index or

item-endorsement index

Answer 23

Just a proportion. correct/total number of people

Question 24

What’s the ideal item difficulty? What should the range be?

Answer 24

.3 to .8, ideal = .5 for discrimination.

(basis of chance + 1) / 2

Question 25

Item-reliability index

Answer 25

standard deviation multiplied by correlation of item score and total test score. Internal consistency.

Question 26

Item-variability index

Answer 26

Degree to which a test measures what it purports to measure.

Question 27

Item-discrimination index

Answer 27

lowercase "d". Difference between high scorers (upper 25-33%) answering an item correctly and proportion of low scorers (lower 25-33%) answering it correctly. Negative d is bad. Means low scorers answer it more correctly than high scorers.

Question 28

Item-characteristic curve

Answer 28

A graphic representation of item difficulty and discrimination. Probability of correct (y) and Ability (x)

Question 29

What are biased test items?

Answer 29

Ones that prohibit item fairness by favoring a group.

Question 30

How do ICCs helpp identify bias?

Answer 30

Different ICCs for different groups even if totals are the same.

Question 31

How should item analysis people deal with speed tests? What's the problem with analyzing speed tests?

Answer 31

Items at the end might be rushed or have no good results, leading to wrong interpretations of analyses. Adminster the test with lots of time for item-analyses.

Question 32

"think aloud" test administration

Answer 32

Testtaker thinks through his thought process and says out loud. Qualitative research. Sees if they using right line of thought.

Question 33

Sensitivity review

Answer 33

Examine fairness of a test and look for stereotypes, offensive language, etc.

Question 34

Cross-validation

Answer 34

revalidation of a test on a new sample.

Question 35

Validity shrinkage

Answer 35

Items for a final version of a test will have lower item validity.

Question 36

Co-validation Co-norming

Answer 36

Using 2+ tests for same sample of testtakers. Making or revising existing norms with 2+ tests for same sample.

Question 37

Anchor Protocol

Answer 37

A test protocol scored by a highly authoritative scorer