CH8: Test Development

Question 1

what is an umbrella term for all that goes into the process of creating a test?

Answer 1

test development

Question 2

what are the 5 stages of developing a test?

Answer 2

1. Test Conceptualization
2. Test Construction
3. Test Tryout
4. Item Analysis
5. Test Revision

Question 2

this stage of test development entails writing test items (or rewriting or revising existing items), as well as formatting items, setting scoring rules, and otherwise designing and building a test

Answer 2

Test Construction

Question 2

this stage of test development involves statistical procedures employed to assist in making judgments about which items are good as they are, which items need to be revised, and which items should be discarded

Answer 2

item analysis

Question 2

in this type of test, items should measure whether test-takers meet specific criteria, regardless of their position relative to others.

Success is defined by meeting set criteria, not by ranking.

Answer 2

criterion-referenced tests

Question 2

this stage of test development refers to action taken to modify a test’s content or format for the purpose of improving the test’s effectiveness as a tool of measurement

Answer 2

test revision

Question 3

what are the key considerations when developing a test?

Answer 3

1. Purpose
2. Need and Users
3. Content and Administration
4. Test Forms
5. Training
6. Responses and Impact
7. Score Interpretation

Question 3

in this type of test, items are deemed “good” if high scorers answer correctly and low scorers answer incorrectly.

Answer 3

norm-referenced tests

Question 4

this term refers to the preliminary research and testing around the creation of a test prototype

Answer 4

pilot work

Question 5

what is the purpose of pilot work/studies?

Answer 5

pilot studies help evaluate the potential test items to determine their suitability for the final version of the test

Question 6

this term refers to the process of setting rules for assigning numbers or indices to measure different amounts of a trait, attribute, or characteristic.

Answer 6

scaling

Question 7

what are the different types of scaling methods?

Answer 7

1. Rating Scale
2. Likert Scale
3. Method of Paired Comparisons
4. Comparative Scaling
5. Guttman Scale
6. Direct Estimation
7. Thurstone Scale

Question 8

this type of scale is a summative scale where test-takers rate the strength of a trait, attitude, or emotion

Answer 8

rating scale

Question 9

this type of scale is commonly used in psychology for attitudes, providing options on a continuum

Answer 9

likert scale

Question 9

what does a unidimensional rating mean?

Answer 9

the scale only measures one underlying dimension

Question 10

what does a multidimensional rating mean?

Answer 10

the scale measures multiple dimensions

Question 10

this type of scale produces ordinal data by comparing stimuli

Answer 10

method of paired comparisons

Question 11

this type of scale involves methods like equal-appearing intervals, where responses are directly rated without conversion

Answer 11

Direct Estimation

Question 11

this type of scale involves judgement of stimuli in relation to others on the scale

Answer 11

comparative scaling

Question 12

this type of scale yields ordinal-level measurements, using scalogram analysis to map responses

Answer 12

Guttman Scale

Question 13

what are the preliminary questions you should think about when writing items?

Answer 13

1. What content areas should be covered?
2. Which item formats are best?
3. How many items are needed overall and per content area?

Question 14

this term refers to the collection of potential test items that will be refined for the final test.

Answer 14

Item Pool

Question 15

what are the two types of item formats?

Answer 15

1. Selected-Response Format
2. Constructed-Response Format

Question 16

this type of item-format requires choosing an answer from given options
(e.g., multiple-choice, true-false)

Answer 16

selected-response format

Question 17

this type of items include a “stem,” a correct option, and distractors.

Answer 17

multiple-choice items

Question 18

this type of items include two possible responses, such as true/false

Answer 18

binary-choice item

Question 19

this type of item involve matching premises with correct responses

Answer 19

matching items

Question 19

this term refers to interactive testing where item selection depends on previous answers, reducing floor and ceiling effects

Answer 19

computerized adaptive testing (CAT)

Question 19

this type of item format requires creating or supplying an answer

Answer 19

constructed-response format

Question 20

this term refers to a large, accessible database of questions for computerized tests

Answer 20

item bank

Question 21

this effect limits distinguishing low-ability test-takers

Answer 21

floor effect

Question 22

this effect limits distinguishing high-ability test-takers.

Answer 22

ceiling effect

Question 23

this term refers to tailoring item content and order based on previous responses.

Answer 23

item branching

Question 24

this type of scoring involves how higher scores indicate higher ability or presence of the trait being measured.

Answer 24

cumulative model

Question 25

this type of scoring involves responses that assign test-takers to specific classes or categories

Answer 25

class scoring (category scoring)

Question 26

this type of scoring compares scores within different scales of the same test, focusing on internal comparisons rather than across individuals

Answer 26

ipsative scoring

Question 27

what is the rule of thumb for test tryout?

Answer 27

Ideally, each test item should have a minimum of 5 subjects, with 10 subjects per item being preferable

Question 27

what are the subject requirements for test tryout?

Answer 27

1. Rule of Thumb 2. Rationale

Question 28

what is a needed characteristic of a good item?

Answer 28

Discriminative Ability

Question 29

what does discriminative ability refer to?

Answer 29

it refers to how a quality test item effectively differentiates between high and low scorers, with high scorers likely to answer correctly or as expected

Question 30

what are the tools for item analysis?

Answer 30

1. Item Difficulty Index 2. Item Reliability Index 3. Item Validity Index 4. Item Discrimination Index

Question 31

this tool of item analysis measures the proportion of test-takers who answered an item correctly, denoted as p (e.g., p1 for item 1). A higher p indicates an easier item

Answer 31

item difficulty index

Question 32

this tool for item analysis indicates internal consistency of a test. Calculated as the product of the item-score standard deviation (s) and the correlation (r) between the item score and the total test score.

Answer 32

item reliability index

Question 33

this tool of item analysis reflects how well an item measures what it is intended to measure, determined by the item-score standard deviation and the correlation between the item score and the criterion score

Answer 33

item validity index

Question 34

this tool of item analysis measures how well an item differentiates between high and low scorers on the test, denoted by d.

Answer 34

item discrimination index

Question 34

this term refers to the graphic representations that illustrate item difficulty and discrimination. A steeper slope indicates greater discrimination ability of the item

Answer 34

item-characteristic curves

Question 35

this term refers to whether a test item is biased against certain groups when controlling for group ability

Answer 35

item fairness

Question 35

this term refers to something that identifies bias when item characteristic curves differ between groups, indicating that performance varies for equivalent ability levels

Answer 35

differential item functioning

Question 36

what is the threshold for item discrimination?

Answer 36

greater than 0.15

Question 36

what is the threshold for item difficulty?

Answer 36

0.20-0.80

Question 36

what is the threshold for item reliability?

Answer 36

greater than 0.75

Question 37

this stage of test development refers to the process of defining the construct to be measured and setting its parameters. This process also includes preliminary decisions about who, what, when, where, and why aspects of the test

Answer 37

test conceptualization

Question 37

what are the two ways of defining a construct in test development?

Answer 37

1. a priori assumption 2. pilot work (qualitative preliminary work)

Question 38

this scaling method presents two items at a time, asking respondents to choose one based on a specific criterion Example: Assessing preferences for different stress-reduction techniques by presenting pairs such as "meditation vs. exercise" and asking individuals to choose which they find more effective.

Answer 38

paired comparison scale

Question 39

this scaling method involves rating an item relative to a benchmark or another item on the scale. Evaluating a patient's anxiety level by comparing it to their typical anxiety level before treatment, using ratings such as "much lower," "slightly lower," "the same," "slightly higher," or "much higher."

Answer 39

comparative scale

Question 40

this scaling method uses categories or labels to classify responses. It is nominal, meaning the categories have no inherent order or numerical value.

Answer 40

categorical scale

Question 41

this scaling method measures the extent to which individuals possess a particular attitude or characteristic. The items are arranged in a cumulative order, such that agreeing with a higher-level item implies agreement with all lower-level items. Example: Measuring social tolerance with items ranging from "willing to live in the same country" to "willing to marry" someone from a different racial group.

Answer 41

Guttman Scale

Question 42

what scaling methods are the most flexible?

Answer 42

1. Rating Scale 2. Likert Scale

Question 42

this scaling method uses equal-appearing intervals to measure attitudes. Experts assign values to statements, and respondents select the statements they agree with. The average value of these statements becomes the respondent's score. Example: Measuring attitudes toward psychotherapy where experts rate a set of statements. Respondents then select statements that align with their views, and their attitude score is the average scale value of the statements they endorse.

Answer 42

Thurstone Scale

Question 43

what scaling methods are usually associated with behaviors?

Answer 43

1. Paired Comparison 2. Comparative Scale 3. Categorical

Question 44

what scaling methods are associated with attitudes?

Answer 44

1. Guttman Scale 2. Thurstone Scale

Question 45

what are the designations for the item difficulty index?

Answer 45

% correct 0-20: Very Difficult 21-60: Difficult 61-90: Moderately Difficult 91-100: Easy

Question 46

what are the levels/designations for the item discrimination index?

Answer 46

<0.19 = Poor item, should be eliminated or needed to be revised 0.20-0.29 = Marginal item, needs some revision 0.30-0.39 = Reasonably good item but possibly for improvement >0.40 = very good item

Question 47

what is an unacceptable range for item difficult index?

Answer 47

<0.20 or >0.80

Question 48

what are the designations for the item reliability and validity index?

Answer 48

Threshold : >0.75 (r) and 0.75 (v) Unacceptable: <0.20 Marginal: 0.21-0.40 Reasonable: 0.41-0.74 Ideal: >0.74

Question 49

this term refers to the revalidation of a test on a sample other than those on whom it was first found to be a predictor [validity shrinkage]

Answer 49

cross-validation

Question 50

this term refers to the validation process conducted on two or more tests using the same sample of test-takers.

Answer 50

co-validation

Question 51

this term refers to a test protocol scored by a highly authoritative scorer designed as a model for scoring and resolving discrepancies

Answer 51

anchor protocol

Question 52

this refers to a neurological disorder marked by involuntary episodes of laughing or crying, often without an appropriate trigger.

Answer 52

Pseudobulbar Affect (PBA)