Test Development

Question 1

what is an umbrella term for all that goes into the process of creating a test?

Answer 1

test development

Question 2

what are the 5 stages of developing a test?

Answer 2

1. Test Conceptualization
2. Test Construction
3. Test Tryout
4. Item Analysis
5. Test Revision

Question 3

this stage of test development involves statistical procedures employed to assist in making judgments about which items are good as they are, which items need to be revised, and which items should be discarded

Answer 3

item analysis

Question 4

this stage of test development entails writing test items (or rewriting or revising existing items), as well as formatting items, setting scoring rules, and otherwise designing and building a test

Answer 4

Test Construction

Question 5

in this type of test, items should measure whether test-takers meet specific criteria, regardless of their position relative to others.

Success is defined by meeting set criteria, not by ranking.

Answer 5

criterion-referenced tests

Question 6

this stage of test development refers to action taken to modify a test’s content or format for the purpose of improving the test’s effectiveness as a tool of measurement

Answer 6

test revision

Question 7

in this type of test, items are deemed “good” if high scorers answer correctly and low scorers answer incorrectly.

Answer 7

norm-referenced tests

Question 8

this term refers to the preliminary research and testing around the creation of a test prototype

Answer 8

pilot work

Question 9

what is the purpose of pilot work/studies?

Answer 9

pilot studies help evaluate the potential test items to determine their suitability for the final version of the test

Question 9

this term refers to the process of setting rules for assigning numbers or indices to measure different amounts of a trait, attribute, or characteristic.

Answer 9

scaling

Question 10

what are the different types of scaling methods?

Answer 10

1. Rating Scale
2. Likert Scale
3. Method of Paired Comparisons
4. Comparative Scaling
5. Categorical Scale
6. Guttman Scale
7. Thurstone Scale

Question 10

this type of scale is a summative scale where test-takers rate the strength of a trait, attitude, or emotion

Answer 10

rating scale

Question 11

this type of scale is commonly used in psychology for attitudes, providing options on a continuum

by asking respondents to rate their agreement with a statement

Answer 11

likert scale

Question 12

what does a unidimensional rating mean?

Answer 12

the scale only measures one underlying dimension

Question 13

what does a multidimensional rating mean?

Answer 13

the scale measures multiple dimensions

Question 14

this type of scale produces ordinal data by comparing stimuli

presents two items at time, asking respondents to choose one based on a specific criterion

Answer 14

method of paired comparisons/paired comparison scale

Question 15

this type of scale involves judgement of stimuli in relation to others on the scale

rating an item relative to a benchmark or another item on the scale

Answer 15

comparative scaling

Question 15

this term refers to the collection of potential test items that will be refined for the final test

Answer 15

Item Pool

Question 16

this type of item-format requires choosing an answer from given options
(e.g., multiple-choice, true-false)

Answer 16

selected-response format

Question 16

what are the two types of item formats?

Answer 16

1. Selected-Response Format
2. Constructed-Response Format

Question 17

this type of items include a “stem,” a correct option, and distractors.

Answer 17

multiple-choice items

Question 18

this type of items include two possible responses, such as true/false

Answer 18

binary-choice item

Question 18

this type of item involves matching premises with correct responses

Answer 18

matching items

Question 19

this term refers to interactive testing where item selection depends on previous answers, reducing floor and ceiling effects

Answer 19

computerized adaptive testing (CAT)

Question 20

this type of item format requires creating or supplying an answer

Answer 20

constructed-response format

Question 21

this term refers to a large, accessible database of questions for computerized tests

Answer 21

item bank

Question 22

this effect limits distinguishing low-ability test-takers

Answer 22

floor effect

Question 23

this term refers to tailoring item content and order based on previous responses.

Answer 23

item branching

Question 23

this effect limits distinguishing high-ability test-takers.

Answer 23

ceiling effect

Question 23

this type of scoring involves how higher scores indicate higher ability or presence of the trait being measured.

Answer 23

cumulative model

Question 24

what does discriminative ability refer to?

Answer 24

it refers to how a quality test item effectively differentiates between high and low scorers, with high scorers likely to answer correctly or as expected

Question 24

this tool of item analysis measures the proportion of test-takers who answered an item correctly, denoted as p (e.g., p1 for item 1). A higher p indicates an easier item

Answer 24

item difficulty index

Question 24

this type of scoring compares scores within different scales of the same test, focusing on internal comparisons rather than across individuals

Answer 24

ipsative scoring

Question 25

this type of scoring involves responses that assign test-takers to specific classes or categories

Answer 25

class scoring (category scoring)

Question 25

what is a needed characteristic of a good item?

Answer 25

Discriminative Ability

Question 25

what are the tools for item analysis?

Answer 25

1. Item Difficulty Index 2. Item Reliability Index 3. Item Validity Index 4. Item Discrimination Index

Question 26

this tool for item analysis indicates internal consistency of a test. Calculated as the product of the item-score standard deviation (s) and the correlation (r) between the item score and the total test score.

Answer 26

item reliability index

Question 27

this tool of item analysis reflects how well an item measures what it is intended to measure, determined by the item-score standard deviation and the correlation between the item score and the criterion score

Answer 27

item validity index

Question 28

this tool of item analysis measures how well an item differentiates between high and low scorers on the test, denoted by d.

Answer 28

item discrimination index

Question 29

this term refers to whether a test item is biased against certain groups when controlling for group ability

Answer 29

item fairness

Question 29

this term refers to the graphic representations that illustrate item difficulty and discrimination. A steeper slope indicates greater discrimination ability of the item

Answer 29

item-characteristic curves

Question 30

what is the threshold for item discrimination?

Answer 30

greater than 0.19

Question 31

what is the threshold for item difficulty?

Answer 31

0.20-0.80

Question 31

what is the threshold for item reliability?

Answer 31

greater than 0.75

Question 32

what are the two ways of defining a construct in test development?

Answer 32

1. a priori assumption 2. pilot work (qualitative preliminary work)

Question 32

this stage of test development refers to the process of defining the construct to be measured and setting its parameters. This process also includes preliminary decisions about who, what, when, where, and why aspects of the test

Answer 32

test conceptualization

Question 33

this scaling method involves rating an item relative to a benchmark or another item on the scale. Evaluating a patient's anxiety level by comparing it to their typical anxiety level before treatment, using ratings such as "much lower," "slightly lower," "the same," "slightly higher," or "much higher."

Answer 33

comparative scale

Question 33

this scaling method presents two items at a time, asking respondents to choose one based on a specific criterion Example: Assessing preferences for different stress-reduction techniques by presenting pairs such as "meditation vs. exercise" and asking individuals to choose which they find more effective.

Answer 33

paired comparison scale

Question 34

this scaling method uses equal-appearing intervals to measure attitudes. Experts assign values to statements, and respondents select the statements they agree with. The average value of these statements becomes the respondent's score. Example: Measuring attitudes toward psychotherapy where experts rate a set of statements. Respondents then select statements that align with their views, and their attitude score is the average scale value of the statements they endorse.

Answer 34

Thurstone Scale

Question 34

this scaling method measures the extent to which individuals possess a particular attitude or characteristic. The items are arranged in a cumulative order, such that agreeing with a higher-level item implies agreement with all lower-level items. Example: Measuring social tolerance with items ranging from "willing to live in the same country" to "willing to marry" someone from a different racial group.

Answer 34

Guttman Scale

Question 34

what scaling methods are the most flexible?

Answer 34

1. Rating Scale 2. Likert Scale

Question 35

this scaling method uses categories or labels to classify responses. It is nominal, meaning the categories have no inherent order or numerical value.

Answer 35

categorical scale

Question 36

what scaling methods are usually associated with behaviors?

Answer 36

1. Paired Comparison 2. Comparative Scale 3. Categorical

Question 36

what scaling methods are associated with attitudes?

Answer 36

1. Guttman Scale 2. Thurstone Scale

Question 37

what are the designations for the item difficulty index?

Answer 37

% correct 0-20: Very Difficult 21-60: Difficult 61-90: Moderately Difficult 91-100: Easy

Question 38

what are the levels/designations for the item discrimination index?

Answer 38

<0.19 = Poor item, should be eliminated or needed to be revised 0.20-0.29 = Marginal item, needs some revision 0.30-0.39 = Reasonably good item but possibly for improvement >0.40 = very good item >0.50 = ideal

Question 38

what is an unacceptable range for item difficulty index?

Answer 38

<0.20 or >0.80

Question 39

what are the designations for the item reliability and validity index?

Answer 39

Threshold : >0.75 (r) and 0.75 (v) Unacceptable: <0.20 Marginal: 0.21-0.40 Reasonable: 0.41-0.74 Ideal: >0.74

Question 40

this term refers to the revalidation of a test on a sample other than those on whom it was first found to be a predictor [validity shrinkage]

Answer 40

cross-validation

Question 40

this term refers to a test protocol scored by a highly authoritative scorer designed as a model for scoring and resolving discrepancies

Answer 40

anchor protocol

Question 40

this term refers to the validation process conducted on two or more tests using the same sample of test-takers.

Answer 40

co-validation

Question 41

what does it mean when there are negative item value discrimination indices?

Answer 41

the LG is better than the UG in that item

Question 41

this refers to a neurological disorder marked by involuntary episodes of laughing or crying, often without an appropriate trigger.

Answer 41

Pseudobulbar Affect (PBA)