CHAPTER 8: TEST DEVELOPMENT

Question 1

It is an umbrella term for all that goes into the process of creating a test.

Answer 1

Test Development

Question 2

It is where the idea is conceived.

Answer 2

Test Conceptualization

Question 3

A stage in the test development process that entails writing test items (or re-writing or revising existing items), formatting items, setting scoring rules, and otherwise designing and building a test.

Answer 3

Test Construction

Question 4

It is done after the form of the test that has been developed, and it is administered to a representative sample of test takers under conditions that simulate the conditions that the final version of the test.

Answer 4

Test Tryout

Question 5

A process where statistical procedures are employed to assist in making judgments about which items are good as they are, which items need to be revised, and which items should be discarded. The analysis of the test’s items may include analyses of item reliability, item validity, and item discrimination. Depending on the type of test, item-difficulty level may be analyzed as well.

Answer 5

Item Analysis

Question 6

It refers to action taken to modify a test’s content or format for the purpose of improving the test’s effectiveness as a tool of measurement.

Answer 6

Test Revision

Question 7

The development of test items differs based on whether a test is norm-referenced or criterion-referenced. In norm-referenced tests, good items are those that effectively differentiate between high and low performers, aiming to rank individuals relative to each other. In contrast, criterion-referenced tests are designed to determine whether an individual has mastered specific knowledge or skills, regardless of how others perform. Item development in criterion-referenced tests focuses on clearly measuring mastery of defined criteria, often used in contexts like licensing exams or educational assessments, where competence is the goal, not comparison.

Answer 7

Norm-Referenced vs. Criterion-Referenced Tests: Item Development Issues

Question 8

It is also known as, pilot study, and pilot research, which refers to the preliminary research surrounding the creation of a prototype of the test.

Answer 8

Pilot Work

Question 9

It may be defined as the process of setting rules for assigning numbers in measurement. It is the process by which a measuring device is designed and calibrated and by which numbers (or other indices)—scale values—are assigned to different amounts of the trait, attribute, or characteristic being measured.

Answer 9

Scaling

Question 10

He significantly advanced the field of psychological measurement by introducing methodologically rigorous scaling methods. He was one of the first to adapt psychophysical techniques to assess psychological constructs like attitudes and values.

Answer 10

L. L. Thurstone

Question 11

It is a procedure for obtaining a measure of item difficulty across samples of test takers who vary in ability.

Answer 11

The notion of absolute scaling

Question 12

Categorizes data without any order (e.g., gender, diagnosis type).

Answer 12

Nominal scale

Question 13

Ranks data in order, but intervals between ranks are not equal (e.g., class ranking).

Answer 13

Ordinal scale

Question 14

Equal intervals between points, but no true zero (e.g., IQ scores).

Answer 14

Interval scale

Question 15

Equal intervals with an absolute zero point (e.g., reaction time, weight).

Answer 15

Ratio scale

Question 16

Measures performance in relation to age (e.g., developmental milestones).

Answer 16

Age-based scale

Question 17

Measures performance in relation to educational grade level (e.g., reading level).

Answer 17

Grade-based scale

Question 18

Raw scores are converted into a 1–9 scale, with a mean of 5 and a standard deviation of ~2.

Answer 18

Stanine scale

Question 19

Measures a single trait or construct (e.g., depression).

Answer 19

Unidimensional scale

Question 20

Measures multiple traits or constructs (e.g., Big Five personality traits).

Answer 20

Multidimensional scale

Question 21

Requires respondents to compare items or choose between options (e.g., forced-choice format).

Answer 21

Comparative scale

Question 22

Assigns responses to distinct, labeled categories (e.g., yes/no, agree/disagree).

Answer 22

Categorical scale

Question 23

It can be defined as a grouping of words, statements, or symbols on which judgments of the strength of a particular trait, attitude, or emotion are indicated by the test taker.

Answer 23

Rating Scale

Question 24

It was developed to be “a practical means of assessing what people believe, the strength of their convictions, as well as individual differences in moral tolerance”, and it contains 30 items.

Answer 24

Morally Debatable Behaviors Scale–Revised (MDBS-R)

Question 25

A type of scale wherein the final test score is obtained by summing the ratings across all the items.

Answer 25

Summative Scale

Question 26

It is used extensively in psychology, usually to scale attitudes. These scales are relatively easy to construct. Each item presents the test taker with five alternative responses (sometimes seven), usually on an agree–disagree or approve–disapprove continuum.

Answer 26

Likert Scale

Question 27

It is a scaling technique that is used to produce ordinal data, where test takers are presented with pairs of stimuli (e.g., objects, statements, images) and must choose one based on a specific criterion (e.g., preference, agreement, appeal). This method determines the relative ranking of stimuli by analyzing consistent patterns of choice across multiple pairs.

Answer 27

Method of Paired Comparisons

Question 28

It is yet another scaling method that yields ordinal-level measures. Items on it range sequentially from weaker to stronger expressions of the attitude, belief, or feeling being measured.

Answer 28

Guttman Scale

Question 29

It is an item-analysis procedure and approach to test development that involves a graphic mapping of a test taker’s responses. The objective for the developer of a measure of attitudes is to obtain an arrangement of items wherein endorsement of one item automatically connotes endorsement of less extreme positions.

Answer 29

Scalogram Analysis

Question 30

It was developed by Thurstone, that is a scaling technique used to construct interval-level attitude scales.

Answer 30

Method of Equal-appearing Intervals

Question 31

Items presented in a ___________ format require test takers to select a response from a set of alternative responses.

Answer 31

Selected-response Format

Question 32

Items presented in a ___________ format require test takers to supply or create the correct answer, not merely to select it.

Answer 32

Constructed-response Format

Question 33

An item written that has three elements: (1) a stem, (2) a correct alternative or option, and (3) several incorrect alternatives or options, variously referred to as distractors or foils.

Answer 33

Multiple-choice Format

Question 34

It pertains to items that have several incorrect alternatives or options.

Answer 34

Distractors or Foils

Question 35

In this type of item, the test taker is presented with two columns: premises on the left and responses on the right. The test taker’s task is to determine which response is best associated with which premise.

Answer 35

Matching Item

Question 36

It is a multiple-choice item that contains only two possible responses.

Answer 36

Binary-choice Item

Question 37

The most familiar binary-choice item, a type of selected-response item, usually takes the form of a sentence that requires the test taker to indicate whether the statement is or is not a fact.

Answer 37

True-False Item

Question 38

A type of item that requires the examinee to provide a word or phrase that completes a sentence.

Answer 38

Completion Item

Question 39

It requires the test-taker to briefly respond with a few words, a phrase, or one or two sentences.

Answer 39

Short Answer Item

Question 40

It requires the test-taker to write a more extended, organized response that explains, discusses, or analyzes a topic in depth.

Answer 40

Essay Item

Question 41

It is a relatively large and easily accessible collection of test questions.

Answer 41

Item Bank

Question 42

It refers to an interactive, computer-administered test-taking process wherein items presented to the test taker are based in part on test taker’s performance on previous items.

Answer 42

Computerized Adaptive Testing (CAT)

Question 43

It refers to the diminished utility of an assessment tool for distinguishing test takers at the low end of the ability, trait, or other attribute being measured.

Answer 43

Floor Effect

Question 44

It refers to the diminished utility of an assessment tool for distinguishing test takers at the high end of the ability, trait, or other attribute being measured.

Answer 44

Ceiling Effect

Question 45

The ability of the computer to tailor the content and order of presentation of test items on the basis of responses to previous items.

Answer 45

Item Branching

Question 46

The test taker's responses earn credit toward placement in a particular class or category with other test takers whose pattern of responses is presumably similar in some way. This approach is used by some diagnostic systems wherein individuals must exhibit a certain number of symptoms to qualify for a specific diagnosis.

Answer 46

Class scoring (or Category Scoring)

Question 47

The objective of this scoring is comparing a test taker’s score on one scale within a test to another scale within that same test.

Answer 47

Ipsative scoring

Question 48

It is a neurological disorder characterized by frequent and involuntary outbursts of laughing or crying that may or may not be appropriate to the situation.

Answer 48

Pseudobulbar Affect (PBA)

Question 49

It refers to the different types of statistical scrutiny that the test data can potentially undergo at this point.

Answer 49

Item Analysis

Question 50

A statistic that shows how difficult a test item is. It represents the proportion of test-takers who answered the item correctly.

Answer 50

Item-difficulty Index (p-value)

Question 51

It is used in non-cognitive assessments (like personality or attitude scales) to indicate the proportion of respondents who "endorsed" or agreed with a particular statement or item.

Answer 51

Item-endorsement Index

Question 52

It provides an indication of the internal consistency of a test; the higher this index, the greater the test’s internal consistency.

Answer 52

Item-reliability Index

Question 53

A statistical tool useful in determining whether items on a test appear to be measuring the same thing(s).

Answer 53

Factor Analysis

Question 54

It refers to the degree to which items on a test that are supposed to measure the same construct yield similar or consistent responses. It’s a type of internal consistency reliability, and it tells us whether the items are homogeneous—that is, whether they all measure the same underlying trait or concept.

Answer 54

Inter-item Consistency

Question 55

It is a statistic designed to provide an indication of the degree to which a test is measuring what it purports to measure.

Answer 55

Item-validity Index

Question 56

It is a measure of the difference between the proportion of high scorers answering an item correctly and the proportion of low scorers answering the item correctly; the higher the value of d, the greater the number of high scorers answering the item correctly. It is a measure of item discrimination, symbolized by a lowercase italic “d” (d).

Answer 56

Item-discrimination Index

Question 57

It is the process of evaluating how well each response option (correct and incorrect choices, or "distractors") on a multiple-choice test functions.

Answer 57

Analysis of Item Alternatives

Question 58

It is a graphic representation of item difficulty and discrimination.

Answer 58

Item-Characteristic Curves

Question 59

It is one that has eluded any universally acceptable solution.

Answer 59

Guessing

Question 60

It refers to the degree, if any, a test item is biased.

Answer 60

Item Fairness

Question 61

These are techniques of data generation and analysis that rely primarily on verbal rather than mathematical or statistical procedures.

Answer 61

Qualitative Methods

Question 62

It is a general term for various nonstatistical procedures designed to explore how individual test items work.

Answer 62

Qualitative Item Analysis

Question 63

A qualitative research tool designed to shed light on the test taker’s thought processes during the administration of a test. On a one-to-one basis with an examiner, examinees are asked to take a test, thinking aloud as they respond to each item.

Answer 63

“Think Aloud” Test Administration

Question 64

It is a study of test items, typically conducted during the test development process, in which items are examined for fairness to all prospective test takers and for the presence of offensive language, stereotypes, or situations.

Answer 64

Sensitivity Review

Question 65

It refers to the revalidation of a test on a sample of test takers other than those on whom test performance was originally found to be a valid predictor of some criterion.

Answer 65

Cross-validation

Question 66

The decrease in item validities that inevitably occurs after cross-validation of findings.

Answer 66

Validity Shrinkage

Question 67

It may be defined as a test validation process conducted on two or more tests using the same sample of test takers.

Answer 67

Co-validation

Question 68

Used in conjunction with the creation of norms or the revision of existing norms.

Answer 68

Co-norming

Question 69

It is a test protocol scored by a highly authoritative scorer that is designed as a model for scoring and a mechanism for resolving scoring discrepancies.

Answer 69

Anchor Protocol

Question 70

A discrepancy between scoring in an anchor protocol and the scoring of another protocol.

Answer 70

Scoring Drift

Question 71

This phenomenon, wherein an item functions differently in one group of test takers as compared to another group of test takers known to have the same (or similar) level of the underlying trait).

Answer 71

Differential Item Functioning (DIF)

Question 72

In this process, test developers scrutinize group-by-group item response curves, looking for what are termed DIF items.

Answer 72

DIF analysis

Question 73

Are those items that respondents from different groups at the same level of the underlying trait have different probabilities of endorsing as a function of their group membership.

Answer 73

DIF Items